Post New Job

Semper Unitas

Overview

  • Sectors Accounting / Finance
  • Posted Jobs 0
  • Viewed 5
Bottom Promo

Company Description

Open-R1: a Totally Open Reproduction Of DeepSeek-R1

Hey there! This article is an introduction to the job, not a claim that we’ve reproduced R1 yet. We’re building in the open, so as quickly as we have evaluation numbers, we’ll share them. You can follow our development on Hugging Face and GitHub.

True, however it looks like there’s absolutely nothing to be evaluated as of right now. I presume the ultimate goal is to train a new thinking design and after that use the exact same examination metrics as o1 and the DeepSeek-R1.

Well, there ought to be at least some peace of mind check and to make sure the design was trained correctly.

Oh yes, if you are speaking about the evaluation variety of deepseek’s model it’s coming extremely quickly!

As mentioned in the blog site post there is no model called Open-R1 to test at all … not yet anyway. This is a blog detailing that Hugging face will take the R1 Deepseek design, work out how it was developed as detailed in the paper and from what they launched, and then replicate that process.

in truth this is quite much how science works … A develops a strategy, discovery or innovation and it is evaluated by B, C and D to see if it is reproduceable. Thats been the cornerstone of research now for a couple of centuries.

This blog is not saying they have currently done so … Its a blog site laying out an intent to begin training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was only released last week, and even in their paper they outlined the compute hours required. While those are low compute hours for a SOTA design this does not imply you can train said model in a week. I ‘d personally like to be able to train a transformer model in a week, but we may require to wait a while for that level of compute innovation.

So there are no criteria for a model that has not been developed yet right? As described in the blog, and once again in reply to your concern.

However fear not, there is a GitHub Repo currently and factors (hell I may join myself), some prelim work done, and a strategy of attack. A good starting position.

n
@edbeeching
has actually evaluated the launched models already

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so collectively …/ s. This is what the brand-new AI czars are saying

Hi! This blog site post is an intro to the task, not a claim that we’ve replicated R1 yet. We will absolutely share the missing out on piece when we have them, you can expect the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s great and essential to comprehend this incredible buzz that does not have technical understanding and explanation. Science has to do with reproduction, and if they declare to be open, let them fullfill the open part.

Please do publish the training cost.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will indeed be striving to ensure this training dish can work for small language models on consumer hardware because not everybody has a cluster of H100s in the house:-RRB- The tool we used for the images was Excalidraw! https://excalidraw.com

looking forward to it! WTF are your talking about?

should be a joke

It’s really cool to see how the entire open source community comes together!

Ops …

5.5 M is number reporter in the deepseekv3 tech report (simply the training, not the experiment afaik), for R1 hard to estimate tbh but much less than 5.5 M imo

Historically, they have never released code or datasets of their LLM training, so I wouldn’t anticipate this time to be different. If they would launch it that would be fantastic obviously!

Yes obviously!

So generally you’re asking to replace existing censorship with another flavour of censorship?

The code for the designs are inside the model repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and developer of EQUATOR. My research team will be dealing with a paper focused on reproducing certain elements of DeepSeek R1. Our objective is to recreate the cold start and offer your group with a dataset that consists of COT and other strategies to support these efforts. We like to contribute our work to help. Please let me know if you discover this beneficial. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the evaluation numbers? without it you can’t call it reproduction.

8 replies

True, but it seems like there’s nothing to be evaluated as of today. I assume the supreme goal is to train a new reasoning design and then use the exact same examination metrics as o1 and the DeepSeek-R1.

That’s rather interesting, I was asking myself why the concerns the author exposed here are not being asked by others? I think the work they have done is unforgettable however at the very same time I wonder why they would not put these missing out on pieces on if they are supposed to be fully open.
Why even without recreation and understanding of the development they could impact a lot the marketplace in this method?

4 replies

Hi! This post is an introduction to the project, not a claim that we’ve recreated R1 yet. We will absolutely share the missing out on piece when we have them, you can expect the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is good that we see more effort into this direction: more optimization and less brute force.
Also wonder what tool did the author usage for producing step diagram.

2 replies

Excalidraw I’m so delighted that initiative like this already exist, I’m gon na try to contribute:-RRB- 1 reply

looking forward to it! So racist articel

2 replies

WTF are your speaking about?

Awesome to have this open recreation started!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s really cool to see how the entire open source neighborhood comes together!

Does anybody understand the actual training cost of r1? I can’t find it in the paper or the announcement post. Is the 6M expense reported by media simply the number drawn from v3’s training cost?

2 replies

Ops …

Has anybody asked the DeepSeek team to publish their training information and code, or a minimum of share them independently with an independent duplication project like this? Have they rejected such a demand?

A loyal duplication depends upon utilizing the very same dataset and hyperparameters. Otherwise, any significant disparities with the released benchmarks would be difficult to pin down-whether due to training data differences or the duplication approach itself.

1 reply

Historically, they have never ever released code or datasets of their LLM training, so I would not expect this time to be different. If they would release it that would be fantastic naturally!

In the meantime we need to make finest guess price quotes and see if we can get there ourselves.

You provide great replication process of Deepseek reasoning training. I will attempt something similar to it.

This is actually excellent details, can we tweak with specific usage case when code is launched?

1 reply

Yes naturally!

Please consider getting rid of prejudiced, tainted or unaligned training information and make an effort to get rid of copyrighted works from the crawl from intake. This will make the model more functional. If you reused anthropic curation checks, this may likewise assist, remove obviouslybiased information will likely add a lot of value. We don’t desire another tainted, unaligned open source model, right? And no corporate would ever utilize deepseek or a design that recycles it, right?
We value your work for the advantage of humanity, we hope.
Miike C from NJ

1 reply

So generally you’re asking to replace existing censorship with another flavour of censorship?

Can’t wait! Hopefully the model will be uncensored but whatever you can do is alright! Love seeing open source building itself up. I’m not smart adequate to in fact assist however I can contribute moral assistance lol

Hello guys, I am even just trying to discover code for DeepSeek-V2, in order to fully understand multi-head latent attention. You do not seem to have code in Hugging Face even for that. Or am I missing something? Don’t see anything in src/transformers/models. MLA is not appropriately described in their paper, so it would be essential to have code for this.

Bottom Promo
Bottom Promo
Top Promo