We launched our data v2 in public beta at the end of February after launching v1 last May. Users had no longer access to Polygon’s data with their Alpaca API keys and could choose between signing up with Polygon.io for a free month to use their data, use Alpaca’s data v1 or upgrade and starting using Alpaca’s data v2.
The change hasn’t affected Alpaca data v1 and currently there’s no plan to remove the v1 endpoints - your algos can continue to use them. One thing to remember is the v1 market data is sourced from IEX and only covers one exchange. The v2 Unlimited data is directly from SIP feeds and is the same market data as Polygon.io uses.
A little bit about v2 and streaming post-mortem
Alpaca’s new data feed is coming as direct feed from exchanges known as SIP. This gives us ultra low latency and high reliability as the data comes directly into our bare metal servers located in New Jersey next to most of the market participants. We partnered with ICE Data Service to receive the feed and worked on this for many months - now it’s fully up and running. As the data volume can easily exceed one million messages per second, it is delivered with multicast UDP. We built our own stack from scratch to consume this high velocity data reliably, and deployed blade servers physically connected to the data source. The data receiver and streaming endpoint are running on our Kubernetes cluster deployed on these blade servers, which gives us resilient and elastic service capability.
Not long after the launch of the new API in beta, the community reported that there were missing trades and quotes on the live data feeds compared to other data providers.
Engineers dug into this problem right away as the highest priority, and found the packet loss problems on the UDP interface. As you probably know, UDP does not guarantee the delivery of the IP packets as protocol, and it is not an easy problem to make sure packets are not missed. After a lot of day-and-night efforts, we identified the root cause and quickly deployed the fix.
Following this, all the network packet drops disappeared and as a result, the 60s logs show that we have now 0 dropped packets:
$ wc -l alpaca_sorted.log reference_sorted.log
185852 alpaca_sorted.log
181171 reference_sorted.log
We understand it was not ideal that we found this issue after the launch, but at the same time we fully appreciate the community’s reporting as it could have been a lot later to find it without the feedback. While we apologize for the inconvenience you encountered during the testing, we are very thankful and motivated to work with the community to build the best platform ever.
5 years historical data now available
We finished backfilling all trades, quotes and bars for the 2016-2020 period, these are now available through the APIs.
However, the 2021 data still has gaps resulting from the difficulties with the streaming service. We are in the process of backfilling to date and will provide regular updates to the community. Please refer to the most recent status of the available historical data here.
For more information on how to fetch historical data on v2, refer to our resource material here.
Renaming plans
We are listening to developers’ feedback and renaming our current subscription plans to avoid confusion. From now on the Basic plan will be called Free plan and the Pro plan will be called Unlimited plan.
The Free plan is great for new users that are looking to test out the platform with IEX which provides merely ~2.5% market volume.
Our Unlimited plan is great for live trading use – such as building basic algos or creating your own trading app or stock screener. This plan comes with unlimited API usage and access to data from all US exchanges from both SIPs providing 100% market volume.
One more month free beta period!
We are extending our beta period to give you one more month of free Unlimited plan!
If you haven’t yet subscribed, please go to the Market data subscription page to choose your plan.
If you are already subscribed, no action is required - please see your updated next invoice date on your Market data subscription page.
For details on the changes and how to sign up, refer to our previous article here.
