By Jay, from QbitAI
QbitAI | WeChat official account QbitAI
Wait, RSI applied to OCR is this powerful??
Recently, a homegrown open-source model with only 1.3B parameters has crushed document parsing.
First place on OmniDocBench, olmOCR-Bench, and DocParsingBench — first on all three leaderboards.
Let's skip the benchmark talk and go straight to the demo images.
Advanced math hand-written notes in the old-school style, converted to Markdown in one click.
It can even draw chemical structure diagrams...
Ridiculous — basically everything can be parsed.
But what I didn't expect is that it's also insanely fast???
On a single RTX PRO 6000, end-to-end parsing of a 100-page document took only101 seconds。
Uh oh, Company A's efficiency at distilling paper books is about to take off (jk).
It's also gone viral in overseas developer communities, having earned 1.1K+ GitHub Stars.
It even became the hottest item on Twitter's Daily News for a time.
And all of this is inseparable from the sexiest concept in the AI industry right now —
RSI.
This self-evolution-powered new OCR species is called OXR.
Optical "Everything" Recognition.
Global No. 1 on Three Leaderboards
Document parsing is a mountain that sounds like it has no technical depth, yet remains unconquered to this day.
When traditional OCR faces an seemingly unremarkable PDF, it's often a total disaster scene.
Two-column papers get strung together into one line, handwritten formulas become garbled junk, large tables spanning pages collapse structurally, and as for chemical structural formulas, many tools simply discard them as messy lines...
Once this low-quality extraction goes into RAG, no matter how smart the model is, the downstream Tool Calls all go dead.
Now, this barrier blocking AI from obtaining high-quality context has finally been smashed.
- OmniDocBench: 97.18, first place (open-sourced by Shanghai AI Laboratory, covering 10 categories of complex documents)
- olmOCR-Bench: 87.84, first place (launched by the well-known overseas AI institution Ai2, with more than 7,000 hardcore unit-test-level questions)
- DocParsingBench: 87.76, first place (#1, covering 5 major industries and 14 real business scenarios)
An open-source document parsing model called OXR has topped all three world-class leaderboards.
Moreover, the entire evaluation process and code are fully open-sourced and reproducible.
I also gave it a quick hands-on try, and it really is quite different. Finally, it's no longer artificial stupidity.
With previous tools, I basically only expected them to turn images into text, and left the rest to AI's own reasoning and guesswork.
Whereas OXR can swallow in one go13 types of document elements:
tables, formulas, code blocks, images, chemical equations, footnotes, headers, footers, seals and stamps...
Applied in specific scenarios, it looks like this——
Magazines: output structured MD that preserves the complete semantic structure and reading order.
Mathematical formulas: directly converted into standard LaTeX code.
Academic literature?
No more making the LLM laboriously call tools to read them; even two-column layouts don't affect quality.
It can even parse math exam papers???
Looks like GPT will be able to better help elementary school students finish their summer homework from now on o(╥﹏╥)o.
Of course, the most exciting thing is the open-sourcing itself.
You should know that for this kind of vertical small model, open-sourcing means something very different from open-sourcing an LLM.
To be honest, while open-source LLMs are nice, they basically have little to do with ordinary users.
Even a model with only 27B parameters has absurdly high local deployment costs—at the very least you'd need an M5 Pro 64G, and that's not even considering output speed issues.
But this OXR thing,is just 1.3B...
What does that mean?
With just an RTX 3060, you can basically run it at full power and milk it for all it's worth, endlessly.
Overseas developers are also overjoyed.
It's got something going for it.
So, how exactly does a 1.3B document parsing model achieve SOTA accuracy?
The secret is all hidden in post-training——
It actually relies on RSI???
Revealing RSI Training
For a long time, everyone has had a fixed mindset:
For AI to achieve ultimate high precision, you must stack tens-of-billions-level large models and massive amounts of human annotation.
But honestly, in the field of document parsing, annotated data is simply a bottomless pit.
Layouts come in all sorts of strange forms, and professional formulas, code, and charts are intricately complex. Relying purely on human annotation not only costs a fortune, but the error rate of human annotation itself is frighteningly high.
The long-tail Bad Cases in the real world are simply inexhaustible. An engineer painstakingly picks out bad samples, manually corrects them, and retrains; after finally releasing a new version, it still fails when encountering new bizarre layouts.
The speed of data preparation has tightly gripped the model's evolution by the neck.
The team has made fairly bold explorations in this area.
They realized that document parsing has long had a characteristic that is easily overlooked:
Every input hasan objectively determinable standard answer。
Whether the text is correct, whether a table has cross-row or cross-column errors, whether formula symbols are missing—all can be measured precisely.
Since right and wrong can be judged objectively, why wait for humans to clean up the mess?
After doing this thought experiment, OXR made a decision:
In the post-training stage, introduce a document-parsing-oriented Recursive Self-Improvement (RSI) training paradigm.
Without relying on new human hand annotation, let the AI system find its own errors, analyze the causes itself, prepare the next round of training data itself, and iterate and evolve on its own.
This RSI process can be roughly divided into three stages:
1. Prediction & Classification (Prediction&Classification)
OXR first parses massive amounts of unlabeled real-world documents.
Immediately after, the General VLM steps in, combining the original images, parsing results, and judging criteria to split the samples into three categories:
- Correct: completely correct, passed directly.
- Wrong: clearly judged wrong, sent for deep analysis.
- Uncertain: predictions unstable, or at the edge of the distribution, also sent for analysis.
Keeping the Uncertain category is a rather counterintuitive Know-How.
If you only give the model the two black-and-white options of Correct and Wrong, the judging model's hesitation gets forcibly packaged into a definite label, ultimately feeding toxins into the next round of training.
With Uncertain, samples the model is unsure about can be caught by a safety net.
In addition to the VLM, OXR also performsmultiple rollout samplings。
on itself. If a document produces drastically different results across several runs, it shows the model lacks confidence; even if it happens to guess right once, it is thrown directly into the Uncertain set.
Finally, while keeping semantics completely unchanged, the system applies slight image perturbations to these high-disagreement samples (small scaling, compression, slight blurring, or shifting).
If the disagreement expands further after perturbation, it indicates the model is overly sensitive to such harmless visual changes, and this perturbation action is recorded on the spot as a core cause of failure, directly guiding the next round of targeted reinforcement.
2. Attribution & Training Planning (Attribution&Training Planning)
After screening out the wrong and weak samples, you must never feed them directly to the model; you have to figure out exactly where it died.
In this stage, the General VLM acts like an experienced doctor, diagnosing sample by sample: is it a cross-column cell alignment failure, or a cursive handwriting recognition error?
The diagnosed natural-language causes are input into SigLIP2 to extract feature vectors, then clustered through an independent clustering model, forming clear Reason clusters.
For example, a financial statement and a chemistry lab record sheet may look and read nothing alike, but if the attribution clustering finds that both died on some rare cross-row cross-column structure, this gets distilled into a common data construction direction.
This is precision striking—reinforce wherever it's weak.
3. Data Generation (Data Generation)
With clear reason clusters in hand, the generator knows what kind of problems to practice.
OXR's specific approach is: the reason clusters provide the structural difficulties, and the real sample pool provides layout and visual style references.
Each time new data is generated, a layout style is grabbed from the real samples as an anchor, and the structural difficulties to be conquered are naturally woven into it.
The visual vulnerabilities discovered earlier in perturbation testing are also systematically incorporated into the synthetic pipeline, forcing the model to repeatedly desensitize during practice.
Finally, the generated data undergoes rigorous schema validation and flows directly back into training the next generation of OXR.
Once one round is completed, the model's weaknesses are fixed; running a new round of real documents will expose new weaknesses, driving the next round of iteration.
RSI was thus formed.
The data distribution becomes increasingly complete and solid with each iteration.
This is also why OXR can achieve such a high intelligence density.
When data quality reaches a certain level, the model no longer needs to rely on bloated parameter counts to memorize by rote.
A Lightweight, Pluggable Architecture
Having covered data and algorithms, let's move on to the next question.
Accuracy first, I can understand.
But speed also first? What's going on???
On a single RTX PRO 6000, OXR parses a 100-page document end-to-end in just 101.0 seconds, first among open-source solutions.
At first I thought it was because of the small parameter size.
But later I realized that document parsing simply cannot be achieved by a small model alone.
There is a lot of counterintuitive Know-How behind this.
Looking back at the technical evolution of document parsing over the past two years, the industry has actually gone through several stages.
At first, everyone used aPipeline—detection, recognition, and ordering each used a separate proprietary model, making the whole thing extremely bloated, with maintenance costs going through the roof.
Later,VLMbecame popular, and people discovered that all tasks were essentially mappings from images to sequences, so they began trying to use end-to-end large models to handle everything in one go, letting the large model directly output text and layout for the entire page.
While elegant, this approach was slow as a snail, because autoregressive generation of long sequences is extremely compute-intensive.
Later still,PaddleOCR-VLtook a very elegant step:
merging layout analysis (LA) and reading order (RO) into a small model, followed by a recognition model, making it two models and two stages.
This brought the speed of layout detection back to the millisecond level of traditional object detection, directly becoming the industry-standard solution for the past two years.
But is this perfect?
No. Every layout annotation added must be forcibly paired with a reading order annotation, doubling the difficulty of data processing.
After in-depth analysis, the OXR team proposed a completely different task-level restructuring scheme:
a standalone lightweight model handles layout analysis (LA), while the OXR main model handles both content recognition (Recognition) and reading order determination (RO).
That is:
Layout Analysis->Recognition+Reading Order
This seemingly simple split combination brings two enormous qualitative changes in engineering:
First,it completely eliminates the performance loss caused by reading order.
Throughout the entire pipeline, the time spent judging reading order was originally far less than that of text recognition.
After OXR merged reading order judgment into the main model, not only did it not add extra time, but because the main model itself has semantic understanding capability (Content-Aware), the accuracy of reading order judgment improved substantially.
At the same time, the layout analysis model (LA) upstream, having shed the complex burden of ordering, now has an extremely pure task and can directly adopt a lighter, more efficient detection model, causing overall throughput to take off on the spot.
Second,the layout analysis module is independent and fully pluggable.
Different businesses have completely conflicting requirements for layout, and this is a long-tail problem:
For a region with mixed text and images, the content team wants to extract it as a whole image, while the knowledge base team wants to pull out the text inside as body content;
For densely arranged four-panel comics, Business A wants to crop them as a whole, while Business B wants to split them into four small images;
……
In short, it is extremely hard to define, and it's impossible to please everyone.
But in OXR's architecture, layout analysis is a completely independent pluggable module——
achieving the greatest freedom of business adaptation at the minimal cost of modification.
If your business has extremely special layout division requirements, for example, if you need to process ancient books or medical documents, you can completely replace it seamlessly with your own privately trained layout analysis model,
while the backend continues to freeload on OXR's powerful content recognition capabilities.
Out-of-the-Box Ecosystem
Of course, since it was born for the developer community, no matter how lavishly a model is praised, if it cannot be deployed, it is useless.
In this regard, the official team provides a free API, with no registration and no need to apply for a Key.
Yes, completely free, pure freeloading...
The official team stated that they will keep maintaining the open-source free service, at least as long as the server load remains within a bearable range.
How could one not stand up and give it a pedal?!
So I hurriedly integrated this API into my own writing APP.
To be honest, when it comes to PDF parsing, the results are far, far better than the previous solutions.
By the way, for those who want to deploy locally, that is no problem either. OXR is fully compatible with mainstream inference frameworks.
You don't need to laboriously recompile vLLM yourself; you can spin up the service with one click by mounting the plugin in the official Docker or CLI.
Currently, OXR's code, weights, and the complete reproduction workflow of the benchmark evaluation have all been made public.
The links are at the end of the article; interested friends can grab them themselves.
Of course, if you have higher requirements, such as processing more formats like Word, PPT, and Excel, or pursuing more complete post-processing customization capabilities, you can also check out the officially maintained commercial-grade engine.
It is called SoMark.
This is also the name of the OXR team.
One More Thing
Finally, I have a few small thoughts I'd like to share with everyone.
Why, today, should we still devote so much effort to stubbornly working on document parsing?
Today, as Agents are comprehensively taking over workflows across all industries, the vast majority of the most core, most professional private domain knowledge that humanity has accumulated over the past decades still lies quietly inside those PDFs, scanned documents, and reports with chaotic formats and complex layouts.
No matter how smart a large language model is, if the input fed to it is a pile of disordered, broken-table character garbage, all it can spit out is hallucinations.
In the past, many people regarded OCR as low-skilled grunt work, but today, from OCR to OXR, its mission has completely changed.
It is the bridge of human-machine interaction, reconstructing all complex multimodal documents in the human world, intact and lossless, into structured assets that are extremely friendly to Agents.
To be honest, seeing a domestic team doing this kind of work close to engineering implementation is actually quite exciting.
Too many people's attention is focused on the Fancy narrative, chasing cutting-edge technology every day, which instead makes us forget something very important——
What do users actually need?
With LLMs having reached the stage they are at now, the answer to this question is definitely not higher model intelligence.
The bigger bottleneck lies in those fields that no one has yet pioneered, the ones that require Getting Hands Dirty.
If this can form a consensus—
infrastructure for Agents,will fall into place piece by piece, at a speed visible to the naked eye.
GitHub: https://github.com/SoMarkAI/OXR
Hugging Face: https://huggingface.co/SoMarkAI/OXR-1.0
Online Demo:
https://free-oxr.somark.cn/
Free API:
https://free-oxr.somark.cn/api/v1
SoMark official website: https://www.somark.cn