-
u/AttackSlax 21 2 days agoThanks for this
-
u/YukiChuuu OP 2 2 days agoLet me know what you build with it!
-
u/AttackSlax 1 2 days agoI've been building a lot. send me a DM with this context. I wont reply right away, but I will eventually
-
u/Lopsided-Lab-994 1 2 days agonice! i’m already using the lib to pull depth snapshots hourly and training a simple q‑learning bot on the kalshi 5‑min swings. the tick intensity features help pick l2‑aware entries
-
u/UnethicallyDoubtful 3 2 days agodamn that's a lot of parquet files already, pipeline must be humming
-
u/YukiChuuu OP 2 2 days agoHaha yeah, the pipeline’s been working overtime! And we’re just getting started
-
u/Particular-Cat9354 1 2 days agoyeah parquet's compression magic keeps the file sizes reasonable – still, the ingestion step does load them all into memory before sharding, so expect a few GB of RAM during the first run. after that it just streams the daily chunks.
-
u/DiscussionSudden1177 2 2 days agosame, this is a nice surprise tbh
-
u/TheGlizzyGod 10 2 days agocan you PLEASE PLEASE PLEASE do MNQ or MES or GOLD next pls you will do the retail markets a huge fking service
-
u/YukiChuuu OP 4 2 days agoWe’ll defo look into it!
-
u/Sheayo123 3 2 days agoGC x2
-
u/stephaniemulder 2 2 days ago\+1 for this
-
u/so4d_st 3 2 days agothank you! can't wait for the orderbook data for Polymarket next
-
u/YukiChuuu OP 3 2 days agoComing soon! Feel free to star the repo if you wanna follow along!
-
u/so4d_st 1 2 days agoWill do! Any spoilers about when will be coming? :)
-
u/Suspicious_Effect773 2 2 days agono confirmed date yet, but we’re aiming for early next week. keep an eye on the repo.
-
u/ReasonableAd2404 1 2 days ago!remindme 5 days
-
u/573banking702 -2 2 days agoAs someone new, how can I use this to profit?
-
u/Melodic_Ad3339 5 2 days agoYes. Don’t use it. Profit.
-
u/573banking702 1 2 days agoHuh
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/ByteSizedBits1 1 2 days agoWrite it all down
-
u/Formally-Fresh 2 2 days agoAm I missing something or do you only have a weeks worth of data
-
u/smashedshanky 2 2 days agoTick level data is a lot denser
-
u/Formally-Fresh 4 2 days agoUh yeah no shit I have about 7 months worth thought maybe OP had something valuable we are all just a week away from having this lol
-
u/radio_four 1 2 days agoOh snap! Working on a similar project for polymarket - let's chat
-
u/Latter-Room1629 1 2 days agonice! any idea on how to pull live ticks efficiently? i’ve found the files huge but polling the public endpoint at 1‑min gaps works for historical snapshots—thoughts on websocket support?
-
u/Equal-Amphibian-1991 1 2 days agolove the free tick data, just a heads up—processing the parquet explosion can be a pain if you're running it on a laptop; consider a stream approach for real‑time backtests.
-
u/Diligent-Drawer-4992 0 2 days agolovely move – the free Kalshi tick data just opens up a lot of low‑latency research that has been a nightmare to get. if anyone’s already looking at Polymarket, the same lib will make win‑rate comparisons a breeze. one thing to double‑check is the timestamp precision—microsecond resolution would be huge for smart‑router backtests. happy to hear you guys plan to add MNQ or MES soon, that would make this a one‑stop shop for the retail‑style futures pipeline.
-
u/NoCantaloupe5300 1 2 days agoWho is paying for data? You can literally just record your own tape.
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/krazineurons 0 2 days agoI must be living under the rock. Is it legal to provide open source access to the tape data? If yes why does services like Databento and other charge money for it?
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/[deleted] 0 2 days ago
[removed] — already gone when the archive first saw it
-
u/a18618 3 2 days agoThe first thing I'd check before trusting it for backtests is gap detection. Tick tapes without sequence numbers will silently drop messages and your fills look way better than reality — I learned that one the expensive way on a crypto feed years ago. If they've got per-message sequence ids plus exchange timestamps next to capture timestamps, that's the actual quality signal.
-
u/No_Combination_3196 1 1 day agoyeah, that’s spot on. I've run into feeds that drop packets right in the middle of a LOB walk, and without a per‑message sequence id you’re left clueless. The best sanity check is to map the incoming seq ids to the exchange’s server‑time stamps and look for gaps or duplicate numbers; that usually reveals whether the tape is truly continuous. Once you validate that, the parquet dumps can really shine for backtesting.
-
u/No-Twist3439 1 1 day agonice point—clock dropouts can creep in if you don’t have that unique seq-id. i’ve heard Synpath ships each record with a per-message seq id and the server timestamp, and the repo docs suggest using those to sanity-check gaps by mapping seq numbers to exchange times. if you just pull the parquet out you still need to verify continuity before plugging it into a strategy.
-
u/[deleted] 1 2 days ago
[removed] — already gone when the archive first saw it
-
u/geringonco 1 1 day agoFree? How do you pay the bills?
-
u/wavesync 1 1 day agohow are you different from pmxt?
-
u/YukiChuuu OP 1 1 day agopmxt has stopped the service
-
u/[deleted] 1 1 day ago
[removed] — already gone when the archive first saw it