IBKR Quant Blog


1 2 3 4 5 2 12


Quant

How Big Data Has Impacted Finance And Banking


By Datafloq

Datafloq

    Domination of markets everywhere.

 

The data revolution that’s fundamentally reshaped the economy from the top down has overwhelmingly been a net positive, however, and for all the disruption that big data recently introduced to the market, businesses everywhere are coming to love it for the savings it can bring them.

Few sectors have benefited from the rise of big data more than the financial and banking sectors, which are starting to employ data analytics with much more gusto than many other industries. So, how exactly is big data changing the faces of these vital industries, and what can professionals within them do to guarantee their futures in an uncertain time?

Don’t fear big data – exploit it

The first thing that you need to understand about big data is that, when it comes to the banking and finance sectors, big data hasn’t been feared and avoided, but rather embraced with zeal. Across the market and a myriad of industries, when working professionals hear terms like artificial intelligencemachine learning, and of course big data analytics, they instantly begin to worry about losing their job to automation. The reality on the ground, however, is that recent tech trends have opened up many more doors than they’ve shut for workers, and that the first step to surviving in the 21st century economy is to exploit these emerging trends, rather than to live in fear of them.

Of course, it goes without saying that even some in the baking and finance sectors often worry about the long-term fate of their occupations, and for good reason; one of the reasons that emerging trends like big data and machine learning are becoming so popular is because of their ability to cut cost while optimizing operations. Despite the anxiety surrounding the rise of smarter machines and the mountains worth of data that power them, the banking and finance industries have learned (to their delight) that these emerging trends prove to be overwhelmingly positive when you consider their full impact across the market.

Primarily, banks have been leveraging big data for success by using it to personalize their relationship with their clients; more so now than ever before, companies that collect massive amounts of data on their customers can build intimate user profiles that help guide executives when the time comes to make important business decisions. The more you understand about how banks vigorously mine data to gain a comprehensive understanding of the people who they’re working with, the better off you’ll be when it comes to determining what they might do with your own data.

In this data, the banks see a veritable goldmine; fraud detection, attracting in wealthy new customers, finding suitable employees and optimizing decision making processes across the board are but some of the things that big data is offering to today’s leading financial institutions.

Finance will soon flourish thanks to big data

Banks aren’t the only ones who are massively benefiting from the rapid rise of big data, of course; the financial sector, including forex factory, stands to seriously gain from this new way of collecting and utilizing data. As a matter of fact, it’s virtually inarguable that almost no sector is better-suited to embrace big data than finance, which will find it easier than ever to comply with regulations, crunch massive sums of info, and predict future trends thanks to recent advances in data analytics.

The positive effects of big data as far as the financial sector goes are almost uncountable thanks to their huge sum; perhaps the most useful facet of the big data revolution to finance remains in how it’s changing fraud detection, however. Consider a world in which financial behemoths can leverage truly awesome sums of information – literally billions of tidbits of data collected from potentially millions of customers and transactions – to determine whether a client is trying to defraud them or not. Thanks to the data revolution, analyzing risk and avoiding hazardous investment opportunities has never been easier.

Of course, not everything surrounding big data is all good news all the time; whether big data analytics is becoming too closely consolidated in a few industries, for instance, is a rapidly growing concern amongst a public that fears nefarious actors may abuse their access to customer’s personal data. Because of these concerns, financial and banking gurus will need to be particularly cautious about how they approach big data analytics, particularly as it pertains to data security and the privacy of the customers who are depending on them.

As the big data revolution continues to ceaselessly churn forth, expect more disruption and innovation to rock the bedrocks of the financial and banking sectors. A new era of big data is only just beginning, and these two industries in particular stand to gain more than almost anyone else. 

 

 

Datafloq offers information, insights and opportunities to drive innovation with big data, blockchain, artificial intelligence and other emerging technologies. You can read high-quality articles, find big data and technology vendors, post jobs, connect with talent, find or publish events and register for our online training. Learn more about Datafloq here: https://datafloq.com

This article is from Datafloq and is being posted with Datafloq’s permission. The views expressed in this article are solely those of the author and/or Datafloq and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17504




Quant

Python Code for Extracting Historical Earnings Data


Learn more about pandas and BeautifulSoup with this post by Milind Paradkar

Python-quant


Earnings announcement events are important for investors and traders. Earnings results of a company tend to determine the future price direction of a stock and thus closely followed and analyzed by them. The period around the date of the earnings announcement sees increased volatility in price and the traded volume of the stocks. Traders can take advantage of this by using different earnings announcement trading strategies.

You can check the “Screener” page on the Quantpedia site where they have listed different earnings announcement trading strategies. The “Screener” page on Quantpedia categorizes hundreds of trading strategies based on different parameters like Period, Instruments, Markets, Keywords etc. Quantpedia has made some of these trading strategies available for free to their users.

Quantpedia

Image Source: Quantpedia

 

Some of the earnings announcement strategies listed on the Quantpedia site include:

To formulate and backtest any earnings announcement strategy, we need the historical dates of earnings release for any given company. Once the dates are extracted, an earnings announcement strategy can be formulated using these dates and other important data like price-volume history, historical performance numbers like sales, profit margin, net margin etc.

Earnings release dates can be obtained from different sources; a financial markets portal being one of such sources. The goal of this article is to extract the historical earnings dates from a popular financial markets portal using Python.

Quantpedia-corporate


Image Source: Corporate Events Table, Quantpedia

 

Python Code for Extracting Historical Earnings Release Dates

Step 1: We first import the required libraries; requests, pandas, and BeautifulSoup. We then copy the URL from the financial market portal for the particular company.

We apply the requests.get function from the Python requests library (which is a simple HTTP library) on the URL which creates a response object (named as “page” in the code). One can check whether we were able to successfully execute the function by checking the status code. The contents of the “page” object can be viewed using the content method.

Python-pandas

Step 2: In this step, we use the BeautifulSoup library for pulling the events table from the HTML. Some basic knowledge of HTML is required in this step. We inspect the HTML page to find the exact location of the events table. We apply the “find_all” method on the respective table class.

Python-BeautifulSoup

Step 3: Note that find_all returns a list. Hence we loop through or use list indexing to extract the required content. We parse the body of the table to grab each row, and extract the values from its columns, and then move on to the next row.

We create two separate empty Python Lists, namely date_list and event_list, which we will populate with the event dates and the event names respectively using the Python For-Loop. The values obtained using the text method is in the Unicode format and we need to convert them to string type.

python-list

 

Step 4: In the final step, we merge the two Python lists into a pandas data frame and select only the “Quarterly Results” and the “Audited Results & Dividend” events. Let us print the earnings release dates of the latest four years.

Puthon-DataFrame

Python-extract-results

One can extract earnings release dates using this Python code and then use other relevant data to build earnings announcement trading strategies.

-------------------------------------

If you want to learn more about quantitative trading strategies using Python, visit QuantInsti website and the educational offerings at their Executive Programme in Algorithmic Trading (EPAT™).

 

Milind Paradkar holds an MBA in Finance from the University of Mumbai and a Bachelor’s degree in Physics from St. Xavier’s College, Mumbai. At QuantInsti®, Milind is involved in creating technical content on Algorithmic & Quantitative trading. Prior to QuantInsti®, Milind had worked at Deutsche Bank as a Senior Analyst where he was involved in the cash flow modeling of structured finance deals covering Asset-backed Securities (ABS) and Collateralized Debt Obligations (CDOs).


This article is from QuantInsti and is being posted with QuantInsti’s permission. The views expressed in this article are solely those of the author and/or QuantInsti and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17405




Quant

Resources for Getting Started with IBKR Python API


Python-IBKR
 

Why choose Python?

Python has been crowned #1 in the “The 2017 Top Programming Languages” by IEEE Spectrum, the world’s largest technical professional organization dedicated to advancing technology for the benefit of humanity.

In addition, StackOverflow, a popular “question and answer site for professional and enthusiast programmers” recently published an article on its Blog noting how popular Python has become. Visit their website to read the “The Incredible Popularity of Python”.

Python language offers robust programming capabilities and is the preferred language for many quant hedge funds and finance professionals. The Python Software Organization offers a Quick Start Guide that walks you through Python installation: https://www.pytho n.org/about/gettingstarted/. You will also need to pick an IDE to code in Python. There are many choices - some of the most popular seem to be Jupyter Notebook, PyCharm, and Spyder.
 

Why choose IBKR Python API?

API users who build their own automated rules-based trading application can use our API solutions. Due to the demand for a framework written in this high performance language, our developers added support for Python in 2017.

If you are a finance professional who’s realized that it is time to start coding, the Python API provides an easier framework compared to Java or C++. Python syntax is more relaxed, and you can practice with the sample code we provide.

To the next generation of traders: college and high school students who have taken classes in Python will appreciate the plethora of complimentary finance educational resources we offer that will go along with the Python reference guide.

 

Getting Started Resources

Here is a short list of resources that will help you get started:

 

  1. Visit IBKR GitHub https://interactivebrokers.github.io/tws-api/ and click on the Python tab.

    Get familiar with the all the resources we offer, including information on Initial setup, Architecture, Streaming Market data, Obtaining Historical Data for backtesting, error handling and message codes, and a list of classes, structs, unions and interfaces with brief descriptions.

    GitHub-IBKR
     
  2. Install
    1. IBKR API from here http://interactivebrokers.github.io/


      Please note that the Python sample Program.py is installed automatically in samples/Python/Testbed directory
       
    2. Trader Workstation Latest from here https://www.interactivebrokers.com/en/index.php?f=16040

  3. Watch "Introduction to the TWS Python API" webinar Recording presented by our API Team:

    www.ibtweet.com/pythonw

    Python-Webinar-Recording

    In this video, the API team will show you:
    1. Capabilities and requirements of the TWS API
    2. API architecture and general programming considerations
    3. Basic flow of an API program as demonstrated by the IB sample Program.py
    4. Current API reference guide
    5. Common API tasks in Python - receiving market data, performing Financial Advisor specific operations


  4. It is important to understand how to establish connectivity because the API requires connectivity via Trader Workstation (TWS) or IB Gateway.

    Learn more with this article "Establishing an API connection"
    https://interactivebrokers.github.io/tws-api/connection.html


  5. Practice

    IBKR reminds you to always test your API components with your simulated trading account or the TWS Demo System before you implement any new API system.
    • Finance professionals
       
    • Students are invited to work with their professors and join the IB Student Trading Lab to practice with a university simulated trading account.

 

The analysis in this article is provided for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such investments. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17467




Quant

Working with Intelligent Machines (4 of 5)


Intelligent-Machines

This is the fourth article in a five part series on AI in Finance. Read Part I here, Part II here and Part III here.

 

Discussions of artificial intelligence usually bring up one of three scenarios in people’s minds:

  1. The Terminator Scenario: AI becomes sentient and overthrows humanity in a violent revolution. Machines are dangerous and our enemy.
  2. The Jetsons Scenario: AI exists in a state of total subservience to humans, like Rosie the robot maid in the Jetsons. Machines are our servants.
  3. The Automation Scenario: AI slowly replaces human workers until there are no jobs left. Society experiences massive upheaval as we adjust to a post-work society. Machines are a replacement for human labor.

Machines will destroy us, serve us, or replace us. Those are the three dominant ways society imagines humanity relating to artificial intelligence.

The Terminator and Jetsons scenarios have been the most widely discussed in fiction and entertainment (hence their names), but it’s the automation scenario that now dominates contemporary thinking around AI. Every day brings another breathless prediction that robots will take all our jobs. One of the most prominent advocates of this position is Tesla (TSLA) CEO Elon Musk, who told the National Governors Association:
 

“There certainly will be job disruption. Because what’s going to happen is robots will be able to do everything better than us. … I mean all of us.”

 

When Musk says “robots will be able to do everything better than us,” he makes two key errors. He overestimates robots, and he underestimates humans. AI will not replace humans, and it won’t destroy us or be totally subservient to us either. Instead, humans and AI will work together [1], combining each other’s strengths and compensating for each other’s weaknesses to create jobs and achieve results that are presently unimaginable.

The Limitations of AI

As we discussed in “Cutting Through the Smoke and Mirrors of AI on Wall Street” (and elaborated on in the second and third articles of this series), AI has a long way to go before it can compete with human intelligence. Machines may be able to defeat humans in games like Chess and Go, but they can’t compete with humans in areas such as logical intuition, much less creativity and innovation. Machines are savants: incredibly skilled at specific tasks but limited in their overall cognition.

The idea that AI will replace all human workers ignores the complexity and variability of most jobs. Sure, you can train a machine to do x, but most jobs require people to seamlessly juggle tasks x, y, and z, often at the same time and in an open environment that is not immutably structured (i.e. a chess board).

Even if AI does advance to match humanity’s level of general intelligence, it will still need to learn how to apply that intelligence. Smart machines need even smarter teachers, and humans are the best teachers around.
 

Figure 1: Teaching Machines

xkcd.com-comic

Image Source: this comic is by xkcd.com. For details visit https://xkcd.com/license.html

 

Machines are undoubtedly superior to humans at a large number of confined or routine tasks, but they will never “be able to do everything better than us.”

Machines Can Empower Us & Make Us More Human

No one understands the pain of being surpassed by a machine better than Gary Kasparov. When the reigning world chess champion was defeated by IBM (IBM) supercomputer Deep Blue in 1997, Newsweek described it as “The Brain’s Last Stand.”

Today, Kasparov sees his defeat differently. He believes the advancement of artificial intelligence represents a boon to humanity, even if machines do cause some disruption to the labor market. The more that AI replaces routine jobs, the more it frees up humans to dream up jobs that could never have existed in the past, as he wrote in an essay last year in the Wall Street Journal.
 

“Machines that replace physical labor have allowed us to focus more on what makes us human: our minds. Intelligent machines will continue that process, taking over the more menial aspects of cognition and elevating our mental lives toward creativity, curiosity, beauty and joy. These are what truly make us human, not any particular activity or skill like swinging a hammer—or even playing chess.”

 

Think of the jobs that exist today that would have been unimaginable 20 years ago. Now think of the jobs that will exist 20 years from now. Just like it would be hard to explain the concept of a social media manager to someone from 1998, so too is it difficult to comprehend the jobs that will exist in 2038. As Box (BOX) CEO Aaron Levie explained on Twitter:
 

“AI can seem dystopian because it’s easier to describe existing jobs disappearing than to imagine industries that never existed appearing.”

 

Humans are wired to fear the unknown. The loss of jobs that already exist to machines frightens us more than the potential jobs those machines will create, even though the new jobs will make use of our talents in ways that are more rewarding and productive.

How We Can Work with Machines

The cycle above—machines taking over human jobs, which then frees those humans to create new jobs—has existed for centuries. Agricultural advancements allowed farmers to move to cities and become artisans and merchants. The Industrial Revolution displaced those artisans and created jobs building railroads and steamships. Throughout history we have made technological advancements that replaced some jobs and created new ones.

What’s different about AI is our relationship to the technology. AI is not a tool where the human is in complete control. AI gets some autonomy. It acts in ways that it’s not explicitly told to and makes recommendations that humans might not anticipate or fully be able to understand. People don’t just need to learn how to use AI, they need to learn how to work cooperatively with it.

Many large financial firms have already shown an understanding that AI is best used as a complement to human labor, not a replacement for it. When BlackRock (BLK) announced last year that it would rely more heavily on algorithms to pick stocks for its funds, much of the news coverage interpreted the move as machines replacing traditional fund managers.

BlackRock, however, insists that’s not the case. Cofounder Rob Kapito outlined the company’s view on AI at the Barclays New Frontier Conference last November:
 

“It’s not going to replace humans. I believe it will be human and machines.”

 

The numbers bear Kapito’s statement out. Even though the company’s shift to more algorithmic funds led to 36 employees leaving, the firm has actually added nearly 700 employees (5% increase) over the past year. It’s not just tech talent either; BlackRock has placed an emphasis on hiring liberal arts majors. As technology changes the finance industry, employers need to start looking for different skillsets.

Research shows that cooperation between humans and machines thrives when humans leverage our uniquely human skills: intuition, pattern recognition, and reading implicit signals. Relationships thrive when both parties have a clear and early understanding of what to expect from each other.

AI has the potential to make us more human, not less, when developed with these goals:

  1. Practice Transparency: In a previous article, we explained the challenges for humans to work with black boxes. Programmers and subject matter experts (SMEs) need to work together to ensure the abstract goals for the machine are properly and clearly expressed in the code. In practice, achieving this goal requires disciplined adherence to clear, documented processes for communication and code writing that both the programmers and SMEs understand.
  2. Understand Strengths & Weaknesses: Machines and humans are most effective together when machines perform tasks that play to their strengths (sorting through large amounts of data, statistical analysis, identifying patterns, etc.) while humans focus on their strengths (putting data in context, interpreting results, understanding priorities, etc.). Get the little things right before adding complexity. Don’t expect too much from your machines or your humans.
  3. Integrate Diverse Minds: If there’s one theme we keep coming back to in this series, it’s the need to integrate technological expertise and with subject matter expertise. The companies that are succeeding with AI enable programmers and SMEs to communicate effectively across all levels of the organization.

These steps can help companies reframe AI as way to make humans more productive rather than as a way to replace them. In the words of famed investor Paul Tudor Jones:
 

“No man is better than a machine, and no machine is better than a man with a machine.”

 

This article is the fourth in a five-part series on the role of AI in finance. The first, “Cutting Through the Smoke and Mirrors of AI on Wall Street” highlights the shortcomings of current AI in finance. The second, “Opening the Black Box: Why AI Needs to Be Transparent” focuses on how transparency is crucial to both developers and users of AI. The third, “AI Has a Big (Data) Problem” details the difficulty machines have in reading large amounts of unstructured data. The final article will focus on how AI can lead to significant benefits for both financial firms and their customers.

 

 

Disclosure: David Trainer and Sam McBride receive no compensation to write about any specific stock, sector, style, or theme.

[1] Harvard Business School features the powerful impact of our research automation technology in the case New Constructs: Disrupting Fundamental Analysis with Robo-Analysts.

Learn more about New Constructs here: https://www.newconstructs.com  

In case you missed New Constructs’ webinar on “Machine Learning for Smarter Investing”, watch the recording on YouTube

 

This article is from New Constructs, LLC and is being posted with New Constructs, LLC’s permission. The views expressed in this article are solely those of the author and/or New Constructs, LLC and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17404




Quant

Getting Started with Neural Networks for Algorithmic Trading - Part III


In this post Kris will continue with the coding and implementation of a Perceptron from Scratch

Let’s now ask our perceptron to learn a slightly more difficult problem. Using the same iris data set, this time we remove the setosa species and train a perceptron to classify virginica and versicolor on the basis of their petal lengths and petal widths. When we plot these species in their feature space, we get this:

Iris

This looks like a slightly more difficult problem, as this time, the difference between the two classifications is not as clear cut. Let’s see how our perceptron performs on this data set.

We’re also going to introduce the concept of the learning rate, which is important to understand if you decide to pursue neural networks beyond the perceptron. The learning rate controls the speed with which weights are adjusted during training. We simply scale the adjustment by the learning rate: a high learning rate means that weights are subject to bigger adjustments. Sometimes this is a good thing, for example, when the weights are far from their optimal values. But sometimes this can cause the weights to oscillate back and forth between two high-error states without ever finding a better solution. In that case, a smaller learning rate is desirable, which can be thought of as fine tuning of the weights.

Finding the best learning rate is largely a trial and error process, but a useful approach is to reduce the learning rate as training proceeds. In the example below, we do that by scaling the learning rate by the inverse of the epoch number.

Here’s a plot of our error rate after training in this manner for 400 epochs:

Epoch

You can see that training proceeds much less smoothly and takes a lot longer than last time, which is a consequence of the classification problem being more difficult. Also note that the error rate is never reduced to zero, that is, the perceptron is never able to perfectly classify this data set. Here’s a plot of the decision boundary, which demonstrates where the perceptron makes the wrong predictions:

Iris-2

Here’s the code for this perceptron:

perceptron

Visit Robot Wealth website to Download the code and data used in this post

 

Learn more about Robot Wealth here: https://robotwealth.com/

This article is from Robot Wealth and is being posted with Robot Wealth’s permission. The views expressed in this article are solely those of the author and/or Robot Wealth and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

 


16849




1 2 3 4 5 2 12

Disclosures

We appreciate your feedback. If you have any questions or comments about IBKR Quant Blog please contact ibkrquant@ibkr.com.

The material (including articles and commentary) provided on IBKR Quant Blog is offered for informational purposes only. The posted material is NOT a recommendation by Interactive Brokers (IB) that you or your clients should contract for the services of or invest with any of the independent advisors or hedge funds or others who may post on IBKR Quant Blog or invest with any advisors or hedge funds. The advisors, hedge funds and other analysts who may post on IBKR Quant Blog are independent of IB and IB does not make any representations or warranties concerning the past or future performance of these advisors, hedge funds and others or the accuracy of the information they provide. Interactive Brokers does not conduct a "suitability review" to make sure the trading of any advisor or hedge fund or other party is suitable for you.

Securities or other financial instruments mentioned in the material posted are not suitable for all investors. The material posted does not take into account your particular investment objectives, financial situations or needs and is not intended as a recommendation to you of any particular securities, financial instruments or strategies. Before making any investment or trade, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice. Past performance is no guarantee of future results.

Any information provided by third parties has been obtained from sources believed to be reliable and accurate; however, IB does not warrant its accuracy and assumes no responsibility for any errors or omissions.

Any information posted by employees of IB or an affiliated company is based upon information that is believed to be reliable. However, neither IB nor its affiliates warrant its completeness, accuracy or adequacy. IB does not make any representations or warranties concerning the past or future performance of any financial instrument. By posting material on IB Quant Blog, IB is not representing that any particular financial instrument or trading strategy is appropriate for you.