Frontier Chips for Frontier AI Labs, with Walter Goodwin, Founder/CEO of Fractile

“Fractal is a chip company.”

Claim: Fractal is a chip company.

“We build very, very fast inference chips for the world's largest models.”

Claim: Fractal builds very fast inference chips for the world's largest models.

“This sort of bet on speed is something which we've had right from the outset.”

Claim: Fractal has focused on speed since the company began.

“Started the company in summer of 2022.”

Claim: Walter Goodwin says he started Fractal in summer 2022.

“And I think at the time we were starting to see two things, you know, one was obviously the arrival of foundation models trained on the internet and generalizing across everything.”

Claim: Walter Goodwin says that when Fractal was starting, foundation models trained on internet data and generalizing across domains were beginning to emerge.

“And the other was, you know, a few kind of wise people saying, you know, we need to find a way to pour more compute into these models at test time. We need to find a way to take what we had for kind of AlphaGo where you have a capable neural network, but then it becomes superhuman as you roll it out at scale.”

Claim: Walter Goodwin says AlphaGo's capable neural network becomes superhuman when rolled out at scale.

“And the other was, you know, a few kind of wise people saying, you know, we need to find a way to pour more compute into these models at test time. We need to find a way to take what we had for kind of AlphaGo where you have a capable neural network,”

Claim: Walter Goodwin says some people advocated allocating more compute to models at test time.

“scale and bring that to kind of language modeling for example and so our big pursuit over the past four years has been to find a way to build chips that will allow for kind of simultaneously taking these huge models and running them much much much faster than the chips of today but do that in a way that kind of scales so scales to models beyond the frontier today scales to extraordinarily long contexts and so that's the the mission that we've been on yeah”

Claim: Walter Goodwin says Fractal's pursuit over the past four years has been to build chips that run very large models much faster than current chips and scale to models beyond today's frontier and to extraordinarily long contexts.

“it's a uh there's a kind of extraordinary zoo now of options available um and if you look across this kind of entire space of kind of ai a6 um one of the things that i think is very striking is”

Claim: Walter Goodwin says the AI accelerator landscape now offers an extraordinary range of options.

“There's a lot of relatively identikit chips out there,”

Claim: Walter Goodwin says many AI accelerator chips are relatively similar in design.

“soft with Maya open AI now with jalapeno these chips that are ultimately developed and delivered in partnership with kind of a relatively small number of what we might call ASIC houses these kind of back-end delivery houses Broadcom is the largest you know two trillion dollar company that help people kind of realize these designs and so when you kind of zoom into this this sort of”

Claim: Walter Goodwin describes Broadcom as the largest ASIC house and a two-trillion-dollar company.

“seeming embarrassment of riches in the spectrum of what exists today, you see that actually there's a lot of similarity across these platforms.”

Claim: Walter Goodwin says AI chip platforms are substantially similar despite the apparent abundance of available options.

“You know, all of these chips, you have HBM, this type of DRAM memory that is, you know, common to NVIDIA GPUs, AMD GPUs and all of these other ASICs. You have the same kinds of bets on tensor cores to do your matrix multiplications, the same.”

Claim: Walter Goodwin says HBM memory is common to NVIDIA GPUs, AMD GPUs, and other ASICs.

“this type of DRAM memory that is, you know, common to NVIDIA GPUs, AMD GPUs and all of these other ASICs. You have the same kinds of bets on tensor cores to do your matrix multiplications, the same.”

Claim: Walter Goodwin says AI chips commonly use tensor cores to perform matrix multiplications.

“advanced packaging with TSMC. And so I think one of the things that we see in the landscape is there's a relative dearth still of kind of efforts that go all the way up and down the silicon stack and try and build kind of fundamentally new capabilities.”

Claim: Walter Goodwin says relatively few efforts span the full silicon stack to create fundamentally new capabilities.

“advanced packaging with TSMC. And so I think one of the things that we see in the landscape is there's a relative dearth still of kind of efforts that go all the way up and down the silicon stack and try and build kind of fundamentally new capabilities.”

Claim: Walter Goodwin says these AI chips use advanced packaging with TSMC.

“There's just not that many teams that have decided we're going to build from, you know, what you might call the kind of, you know, the architecture. The architecture layer, the front-end design layer, which mostly looks like kind of writing code, but also all the way down to what we call kind of physical design, the process technology, the kind of foundry interactions, the stuff that essentially has you flying back and forth to Taiwan or Korea every week.”

Claim: Walter Goodwin says relatively few teams work across the full chip-development stack, from architecture and front-end design through physical design, process technology, and foundry interactions.

“That's something which actually tends to still be left to a relatively small number of companies.”

Claim: Walter Goodwin says foundry-interaction work is generally handled by a relatively small number of companies.

“Yeah, so I think if you're, let's say you're Google and you're designing the TPU, you have a number of people in your team who have a really strong understanding of the workloads that they're trying to accelerate.”

Claim: A team designing Google's TPU includes people who strongly understand the workloads the chip is intended to accelerate.

“You conceive of things like the tensor core, which is a designated circuit dreamt up by an architect that is exceptionally good at matrix multiplications because you have this insight that matrix multiplications are the overwhelming majority of the kind of flop count in these models that you're running.”

Claim: Tensor cores are designated circuits optimized for matrix multiplication, which accounts for the overwhelming majority of the floating-point operation count in the models being run.

“That's kind of the preserve of the architect, I guess, is somebody who really understands the way that chips behave. but also ideally has a very strong understanding of the kind of target workloads.”

Claim: A chip architect should understand chip behavior and ideally understand the workloads the chip is intended to run.

“What you'll then see inside those same organizations is a set of sort of, you know, very smart people that kind of lower that down into, I guess, you know, essentially a kind of circuit level description of how this chip should behave.”

Claim: Chip-design organizations have people who translate the design into a circuit-level description of the chip's intended behavior.

“And so A lot of this is what we call front end design.”

Claim: The work of describing a chip's intended behavior at a high level is called front-end design.

“You know, mechanistically, it still looks like like writing code on a computer and it's still code that is then handed off to a player like Broadcom.”

Claim: Chip front-end design is performed by writing code on a computer, and the code is then handed to a company such as Broadcom.

“And so, you know, that kind of front end design that describes the kind of underlying intent of the chip, the underlying logic of the chip is then transformed ultimately into something that, you know, what gets shipped to TSMC in the end is literally.”

Claim: A chip's front-end design specifies its underlying intent and logic, which is transformed into the final design sent to TSMC.

“There's literally a kind of bitmap, you know, we have this file type you call GDS2, and it's literally, you know, where do I put the metal layers? Where do I put each individual transistor? And so it's a full layout.”

Claim: The final chip layout sent to TSMC is represented as a bitmap-like GDS2 file that specifies the placement of metal layers and individual transistors.

“And so you go through this process of synthesis of that RTL into a kind of set of circuits that then ultimately get laid out.”

Claim: The chip-design process synthesizes RTL into circuits and then lays those circuits out.

“And a lot of the complexity there is around, you know, in Broadcom's case, you know, it will be ownership of... analog ip that is used for chip to chip connectivity and it is this kind of physical placement that is specific to a given process node at tsmc so you know we talk about like three nanometers five nanometers etc um so that piece is still today you know generally it is a kind of outsourced activity for all of these projects”

Claim: Broadcom's chip-design work includes analog IP for chip-to-chip connectivity and physical placement tailored to a particular TSMC process node.

“analog ip that is used for chip to chip connectivity and it is this kind of physical placement that is specific to a given process node at tsmc so you know we talk about like three nanometers five nanometers etc um so that piece is still today you know generally it is a kind of outsourced activity for all of these projects”

Claim: As of the time of the conversation, physical-design work is generally outsourced in these chip projects.

“Yeah, I think there is a you know It's funny there's a lot of industries where there will be a kind of waterfall style approach to doing things and then suddenly you know along comes is a new way of thinking and it's kind of more agile to borrow from like management philosophy right and i think for us as a very full stack company so we have a team that does span you know very deep workload understanding i think we actually try to you know run ahead in many places and look at you know where would we change a model architecture to be more exceptionally aligned with our bet what do we think is like kind of the scaling law vector for the particular bets that we that we are taking.”

Claim: Fractile describes itself as a full-stack company whose team studies workloads and proactively considers model-architecture changes and scaling-law directions aligned with its strategic bets.

“And then we can advocate to our customers and our partners on that.”

Claim: Fractile uses its workload and architecture analysis to advocate its positions to customers and partners.

“And it also helps us inform a set of bets for a chip that isn't going to necessarily exist in volume for one or two years.”

Claim: Fractile's analysis informs chip-design bets for a chip that may not be available in volume for another one or two years.

“That becomes a really important part of the muscle that we have to build.”

Claim: Developing the capability to make informed bets on future chips is an important organizational capability for Fractile.

“it means that inside Fractal, we do have have front-end designers we have our own physical design team we have our own kind of back-end implementation team you know we do our own work on advanced packaging for example um and that's not an enormous head count right i mean fractal today is about 150 people uh we're sort of skinny in in every single one of those uh sectors but what it allows us to do is have this kind of much more agile closed loop um”

Claim: Fractile maintains in-house front-end design, physical design, back-end implementation, and advanced-packaging capabilities.

“have front-end designers we have our own physical design team we have our own kind of back-end implementation team you know we do our own work on advanced packaging for example um and that's not an enormous head count right i mean fractal today is about 150 people uh we're sort of skinny in in every single one of those uh sectors but what it allows us to do is have this kind of much more agile closed loop um”

Claim: Fractile's in-house chip-design capabilities enable a more agile, closed-loop development process.

“have front-end designers we have our own physical design team we have our own kind of back-end implementation team you know we do our own work on advanced packaging for example um and that's not an enormous head count right i mean fractal today is about 150 people uh we're sort of skinny in in every single one of those uh sectors but what it allows us to do is have this kind of much more agile closed loop um”

Claim: At the time of the conversation, Fractile had about 150 employees and relatively small teams in each of its in-house chip-design functions.

“And I think that's becoming increasingly essential as you look at the sort of cadence needs that are set by this industry,”

Claim: An agile, closed-loop chip-development process is becoming increasingly essential to meet the industry's cadence requirements.

“you know, the chasing, you know, perennially of the sort of chasing of the tail of the workloads is, you know, you have to in the AI chip space more than any other kind of chip play before, you have to really place your bets correctly.”

Claim: AI chip companies must anticipate evolving workloads and make their design bets correctly.

“So you have to have a lot of, you know, skill and a lot of luck and you also then need to strike very fast and so this kind of structural setup where we have the entire story in-house uh it's very very different to this kind of i guess prior approach where there's almost like a handoff point and so you get to a certain level and then you hand off to another partner and it's some to some extent you're kind of at the mercy of how that other partner then behaves”

Claim: Success in AI chip design requires both skill and luck, as well as very fast execution.

“fast and so this kind of structural setup where we have the entire story in-house uh it's very very different to this kind of i guess prior approach where there's almost like a handoff point and so you get to a certain level and then you hand off to another partner and it's some to some extent you're kind of at the mercy of how that other partner then behaves”

Claim: Fractile's in-house chip-development structure contrasts with a handoff-based approach in which a company passes work to a partner and becomes partly dependent on that partner's actions.

“just the sheer bet that we were going to slip into a kind of deployment era and the bet on speed.”

Claim: Fractile’s technical strategy was based in part on the bet that AI would enter a deployment era and that inference speed would be important.

“In for a journey, I think for the first kind of two years or so of the company's life, we were working on a little like Grok or Cerebrus, working on an SRAM-based chip.”

Claim: For roughly its first two years, Fractile worked on an SRAM-based chip similar to those associated with Groq or Cerebras.

“So we'd sort of made the observation that SRAM is this super high bandwidth memory. It lives on the same piece of silicon as your logic.”

Claim: SRAM provides very high-bandwidth memory and resides on the same silicon as the logic.

“And so you have this extremely high bandwidth between your compute where you're running the maths as you run these models. and let's say the weights of your model or the kv cache of this model as you're rolling it out um and that's something which then then can drive you to thousands of tokens per second on these language models um but i think one of the things that we started to worry about in kind of towards the end of 2023 and certainly in 2024 was the scalability of this approach um and you know i think there are two things that”

Claim: High-bandwidth access between compute and model weights or KV cache can enable language models to run at thousands of tokens per second.

“and let's say the weights of your model or the kv cache of this model as you're rolling it out um and that's something which then then can drive you to thousands of tokens per second on these language models um but i think one of the things that we started to worry about in kind of towards the end of 2023 and certainly in 2024 was the scalability of this approach um and you know i think there are two things that”

Claim: Toward the end of 2023 and especially in 2024, Fractile became concerned about the scalability of its SRAM-based chip architecture.

“grow with AI today one is obviously the parameters of the model but the other and this was the one that kind of really got us nervous about that architectural approach was you know this kind of growing context length that was becoming more and more part of the story for how we saw these models rolling out and it's that that has taken us over the past couple of years to a kind of what we see as a more exciting bet I guess in some ways which”

Claim: Fractile saw AI workloads growing in model parameter count and context length, with increasing context length particularly raising concerns about the scalability of its architecture.

“Switch is working more closely with memory vendors as well as our sort of logic foundry partners to find ways to get extremely high bandwidth access to higher capacity memories.”

Claim: Fractile’s newer architecture strategy works with memory vendors and logic foundry partners to provide very high-bandwidth access to higher-capacity memory.

“So for the past couple of years, we've been involved in these kind of skunkworks projects to move away from SRAM, you know, looking at how we can get, for instance, much, much, much higher bandwidth to DRAM memories.”

Claim: For the past couple of years, Fractile has pursued projects to move away from SRAM and achieve substantially higher bandwidth to DRAM.

“exciting bet for us because it's allowed us now to put together a platform which we'll be ramping in the second half of next year, which kind of unites, I guess, the scalability of these kind of higher capacity, lower cost DRAM memories that you have on a GPU, on a TPU, with all of the speed advantages that you get from a Grok chip or a Cerebrus chip.”

Claim: Fractile expects to ramp a platform in the second half of the following year that combines higher-capacity, lower-cost DRAM’s scalability with the speed advantages of Groq- or Cerebras-like chips.

“And I think where we see this as especially vital is if you look at where it is that speed really moves the dial today. There is a kind of, there's a version of this which is like a snappier chat bot, but I think this is like, you know, when Henry Ford sort of asked the rhetorical question of what people would have said they wanted and, you know, faster horses rather than a car. The snappier chat bot is kind of the faster horses of kind of fast inference.”

Claim: Walter Goodwin views a snappier chatbot as a limited, “faster horses” application of fast inference rather than its most important opportunity.

“The place where the ability to take a multi-trillion parameter model and run it comfortably at many thousands of tokens per second. Second, in our view, is a kind of fundamental elevator on capability for AI is in taking these kind of very long running agents and making them radically faster.”

Claim: Walter Goodwin believes that running multi-trillion-parameter models at many thousands of tokens per second could radically speed up long-running AI agents and substantially increase AI capability.

“And so that's kind of a, it's a tantalizing mismatch today between the properties of the fast inference chips that we have, which have super high bandwidth memory, but incredibly low capacity.”

Claim: Fast inference chips combine very high memory bandwidth with extremely low memory capacity.

“And if you look at the sort of technical detail of how they actually get employed and deployed today. they're not running long context attention you still flip back to a gpu to do that and so there's this sort of tantalizing mismatch where precisely the area where we are almost able to go and take these things and make them much faster we also don't have that technical capability and so in a sense this is you know like many technical challenges it boils down to a slightly mundane technical observation which is we need chips that have this kind of”

Claim: Current deployments of fast inference chips do not run long-context attention on those chips and instead switch to a GPU for that workload.

“of particular ineffable property which is incredibly high bandwidth to memory so we can load the weights we can load this state you know many thousands of times per second but also have like economical memory because when you look at what it looks like to run inference at data center scale for thousands of users the economics actually collapse down to essentially a kind of cost per gigabyte of the memory that you're employing we're implying.”

Claim: At data-center scale and across thousands of users, inference economics largely reduce to the cost per gigabyte of the memory employed.

“And so that's been the sort of other key focus for us is unlocking this kind of fundamentally new building block, which is finding a path to get aggressively high bandwidth from this kind of, you know, from the world's lower cost memory, DRAM memory.”

Claim: Fractile’s other key technical focus is developing a building block that provides very high bandwidth from lower-cost DRAM.

“but what you want at any given moment in time for an AI chip is you wish that. that you had a chip that was precisely targeting the workload that you care about and in your hand today in volume but if you had that chip uh you know you're going to want something else again in six months time like these workloads move on very fast that's”

Claim: AI workloads change quickly enough that an ideal chip, precisely tailored to the current workload and available in volume today, may be outdated within six months.

“right fast. and so so sanity comes from looking across those models and saying well okay what's what's common to all of these and like Like, thankfully, there's enough that is common.”

Claim: Despite rapid model changes, enough requirements are common across models to guide chip design.

“So, you know, it's always the case that a new LLM desperately wants to have orders of magnitude more bandwidth to memory in order to run faster.”

Claim: New LLMs generally require orders of magnitude more memory bandwidth to run faster.

“It remains the case today that, you know, these LLMs tend to be auto-aggressive, very low batch as you generate text with this sort of fundamental trade-off, again, between kind of throughput efficiency and cost and the speed that you can serve these models at. So there are certain things that these models are.”

Claim: During text generation, LLMs tend to be auto-aggressive and operate at very low batch sizes, creating a trade-off among throughput efficiency, cost, and serving speed.

“unified in kind of crying out for and memory bandwidth is one of them um but then you know as you say there are these evolutions and especially if you look at the kind of frontier of open source chinese models you know the exact nature of an attention mechanism will churn every couple of weeks in terms”

Claim: Memory bandwidth is a shared requirement across AI models.

“unified in kind of crying out for and memory bandwidth is one of them um but then you know as you say there are these evolutions and especially if you look at the kind of frontier of open source chinese models you know the exact nature of an attention mechanism will churn every couple of weeks in terms kind of at the frontier the sparsity the sparsity level of the moes that you have the sparsity on the attention itself um and you know we may be in for like you”

Claim: Among frontier models, attention mechanisms and sparsity levels in MoEs and attention can change every couple of weeks.

“bigger shocks as well than maybe kind of more fundamental shifts and so I think there's this duality around sort of the limits of the physical and the kind of financial world I guess and then these demands of you know workloads that churn at the pace of essentially software where although there is this sort of clear desire to be able to kind of ship new platforms you know aggressively faster I think there are also some sort of fundamental breaks on this.”

Claim: AI workloads can evolve at software-like speed, but physical and financial constraints limit how quickly new chip platforms can be shipped.

“So at Fractal, you know, we're extremely excited about being AI forward in how we do chip design.”

Claim: Fractile is highly enthusiastic about using AI in its chip-design process.

“I think we've seen this ability to own the entire end-to-end gamut of the problem as really enabling a kind of fundamental rethink of how we do things.”

Claim: Fractile's ability to own the entire end-to-end chip-design problem has enabled it to fundamentally rethink its processes.

“like everything I can paralyze becomes very, very. very, very fast and the part that I can't parallelize doesn't.”

Claim: Parallelizable portions of a process can become much faster, but non-parallelizable portions do not speed up in the same way.

“And I think it's similar when you're in a like long chain of organizations that, you know, there isn't necessarily an overwhelming need to or return on radically changing your processes as a front end focused chip design startup adopting AI in aggressive ways to shorten your time. your timelines from being you know maybe 12 months of front-end design work to a tiny fraction of that if there's then going to be kind of the conventional bottlenecks and the conventional pace on the next piece um and i think that's sort of you know at some extent there will always be those bottlenecks you know we all have the same kind of fab cycle times with our foundry partners so so from the moment you send the chip to them to getting it back is you know”

Claim: For a front-end-focused chip-design startup in a long organizational chain, radically shortening front-end design with AI may have limited value if downstream processes remain bottlenecks.

“your timelines from being you know maybe 12 months of front-end design work to a tiny fraction of that if there's then going to be kind of the conventional bottlenecks and the conventional pace on the next piece um and i think that's sort of you know at some extent there will always be those bottlenecks you know we all have the same kind of fab cycle times with our foundry partners so so from the moment you send the chip to them to getting it back is you know you know three to five months even in a kind of super hot lot scenario and so these are the kind of innate latencies i guess in the industry and then when you get that chip back i think it is vital to look at the uh the way that these chips actually become financializable which is that they do have a kind of payoff period you have to have really a kind of three to five year amortization window for that chip to make it a sensible financial decision”

Claim: Foundry fabrication turnaround takes three to five months, even for a super-hot lot.

“you know three to five months even in a kind of super hot lot scenario and so these are the kind of innate latencies i guess in the industry and then when you get that chip back i think it is vital to look at the uh the way that these chips actually become financializable which is that they do have a kind of payoff period you have to have really a kind of three to five year amortization window for that chip to make it a sensible financial decision”

Claim: A chip generally needs a three-to-five-year amortization period to make financial sense.

“um and so there's these sort of two things that are intention and which i i guess i believe two slightly conflicting things at the same time one is there is a huge value in being able to take the overall chip design cycle and compress that and compress that uh there is also i think what we don't lose is the need to place really thoughtful and subtle bets architecturally because the chip you actually you actually end up making does still need to have a three-year plus, you know, useful lifespan.”

Claim: Even with a shorter design cycle, chip designers must make careful architectural bets because the resulting chip needs a useful lifespan exceeding three years.

“um and so there's these sort of two things that are intention and which i i guess i believe two slightly conflicting things at the same time one is there is a huge value in being able to take the overall chip design cycle and compress that and compress that uh there is also i think what we don't lose is the need to place really thoughtful and subtle bets architecturally because the chip you actually”

Claim: Compressing the overall chip-design cycle can create substantial value.

“you know, what it means to be able to over time compress that kind of chip design cycle is essentially that you get to have more shots on goal.”

Claim: A shorter chip-design cycle allows a company to pursue more design opportunities over time.

“That is actually the flagship and this is the one we're going to ramp. But then it's a 12 month, 18 month volume ramp and you're expecting it to have longevity. You're expecting it to actually continue to drive. drive value to your customers and so i think that the the sort of subtlety here is you know i'm not a believer in the idea that we will ultimately get to a place where we are shipping a fundamentally new chip uh you know every few weeks just because we've shortened that down because i think this is a this is a physical world there is a certain amount of constraint on you know data center power there are latencies to installing these things and you need to”

Claim: A flagship chip may take 12 to 18 months to ramp to volume and is expected to keep delivering customer value over time.

“drive value to your customers and so i think that the the sort of subtlety here is you know i'm not a believer in the idea that we will ultimately get to a place where we are shipping a fundamentally new chip uh you know every few weeks just because we've shortened that down because i think this is a this is a physical world there is a certain amount of constraint on you know data center power there are latencies to installing these things and you need to”

Claim: Physical constraints, including data-center power and installation latency, limit how quickly new chips can be deployed.

“drive value to your customers and so i think that the the sort of subtlety here is you know i'm not a believer in the idea that we will ultimately get to a place where we are shipping a fundamentally new chip uh you know every few weeks just because we've shortened that down because i think this is a this is a physical world there is a certain amount of constraint on you know data center power there are latencies to installing these things and you need to”

Claim: Shorter design cycles will not lead to fundamentally new chips being shipped every few weeks, in the speaker's view.

“finance the actual underlying silicon so it needs to have that payoff period but i think what you can get to is a world where the shorter you can make that latency that gap between an observation and realizing that bet in volume that is where there's an you know an enormous amount of value to be captured um and i think it's also an area where you know fractal is a”

Claim: Financing silicon requires the chip to have a sufficient payoff period.

“finance the actual underlying silicon so it needs to have that payoff period but i think what you can get to is a world where the shorter you can make that latency that gap between an observation and realizing that bet in volume that is where there's an you know an enormous amount of value to be captured um and i think it's also an area where you know fractal is a”

Claim: Reducing the time between an observation and realizing the associated chip-design bet at volume can capture substantial value.

“generally across the economy. One is, oh no, we're not going to have as much to do. Maybe there's jobs lost.”

Claim: Rising AI-driven productivity could reduce the amount of work available and lead to job losses across the economy.

“The other is we get to do more things.”

Claim: Rising AI-driven productivity could enable people to do more things.

“standing here from from fractal you know right now we're trying to build a single chip we know that we're up against competitors that have uh you know if you crack open an nvidia system it has anywhere between kind of six and nine custom chips all built by nvidia to come together to build something that is really really potent um so there's already a gap there you know it'd be great to be productive enough that we could start to build our own responses in that same kind of way um and so i think this idea”

Claim: Fractile is trying to build a single chip, whereas an Nvidia system uses roughly six to nine Nvidia-built custom chips working together.

“standing here from from fractal you know right now we're trying to build a single chip we know that we're up against competitors that have uh you know if you crack open an nvidia system it has anywhere between kind of six and nine custom chips all built by nvidia to come together to build something that is really really potent um so there's already a gap there you know it'd be great to be productive enough that we could start to build our own responses in that same kind of way um and so i think this idea”

Claim: Fractile would benefit from having enough productivity to build its own multi-chip systems, similar to Nvidia's approach.

“a kind of rolling frontier of bets that you hope are deeply aligned in which you're ready to trigger the ramp of this is a huge advantage if you can kind of build that machine and build that engine and the six month gap that that might then perennially give you against your competition uh that is the kind of you know wedge that allows you to drive in and we see this in frontier labs right it's the equivalent of the frontier model for the chip space is if you can just”

Claim: A company that maintains a rolling frontier of aligned bets and ramps them when ready may gain a major advantage, including a persistent six-month lead over competitors that creates a competitive wedge.

“Just find a way to structurally carve out a three to six months advantage, you will be winning all of those deployments.”

Claim: A company that structurally secures a three-to-six-month advantage will win all of the relevant deployments.

“a heuristic that has been successful over the past couple of years is to just just always question your sort of logical assumption and then divide it by four on timescale.”

Claim: Over the past couple of years, a useful forecasting heuristic has been to question one's intuitive assumptions and divide the estimated timescale by four.

“So there is a world in which 10 years seems like a very sensible timeframe for that, but I would divide it by four and I probably subtract a bit from that.”

Claim: For the chip-design capability being discussed, a ten-year estimate should be divided by four, with a further small reduction probably warranted.

“You know, I think that we will have a space where there is a kind of prototyping that goes end to end, you know, in the next few years.”

Claim: End-to-end prototyping for the chip-design process will be possible within the next few years, in the speaker's view.

“I think one of the things about chip design that is very interesting is there are still these loops in the middle of it that are kind of conventional solutions to NP-hard problems.”

Claim: The chip-design process still includes intermediate stages that solve NP-hard problems using conventional methods.

“And so you have these sort of, if you really want to output that GDS2 file, today you are still running through a bunch of layout tools that are doing place and route with conventional kind of algorithmic approaches which will run for days on end.”

Claim: Generating a GDS2 file currently requires conventional place-and-route layout tools that may run for days.

“A lot of hard problems in the world, I think, look a bit like this, where there is a lot of thinking that can now be automated. There's a lot of cerebral work. But then there is also, you know, this kind of intrinsic latency to something that you do.”

Claim: Many difficult problems combine automatable cognitive work with intrinsic process latency.

“You know, AI for guiding AI model development today, so the RSI work, you know, yes, we can now have extreme intelligence that is guiding our experiments, but now we're just experiment bottlenecked, you know, we're compute time bottlenecked. And so I think with with chip design,”

Claim: In AI-guided model development, highly capable intelligence can guide experiments, while experiment execution and compute time become bottlenecks.

“it's a little bit like this, maybe to take the kind of, you know, architectures intent through to GDS to question, it's a little bit like the RSI question.”

Claim: The challenge of translating chip architectural intent into a GDS file is analogous to the recursive-self-improvement question.

“In the things that will prevent us getting there extremely fast is the fact that actually in that flow, there are a number of things that are genuinely kind of computationally expensive in a more conventional way.”

Claim: Conventional computationally expensive steps in the chip-design flow will limit how quickly the process can be automated or accelerated.

“I think there's definitely these sort of simulation models, I think driving approximations. you know, all of the models that we see today for kind of finite element analysis and sort of thermal estimation and so on. I think there is a huge return to that kind of work precisely because of this kind of Andals law that as we accelerate the intelligence,”

Claim: Simulation models such as finite-element-analysis and thermal-estimation models can provide approximations for engineering work.

“you know, all of the models that we see today for kind of finite element analysis and sort of thermal estimation and so on. I think there is a huge return to that kind of work precisely because of this kind of Andals law that as we accelerate the intelligence, as we accelerate the period between these experiments, between these trials, becomes much more important to accelerate those trials as well. And so, you know, we are looking internally at,”

Claim: When AI shortens the interval between experiments, speeding up the experiments or trials themselves becomes increasingly important.

“of some of those algorithms and is there a crude approximate way that you could do some of this you know the i think chip design is not going to change in the final sign-off for quite some time it's incredibly valuable to have cadence and synopsis who've been working with tsmc and the other foundries for decades to build the sort of you know the final check mark that says this thing is what we call drc and lvs clean”

Claim: The final sign-off stage of chip design is not expected to change for quite some time.

“of some of those algorithms and is there a crude approximate way that you could do some of this you know the i think chip design is not going to change in the final sign-off for quite some time it's incredibly valuable to have cadence and synopsis who've been working with tsmc and the other foundries for decades to build the sort of you know the final check mark that says this thing is what we call drc and lvs clean”

Claim: Cadence and Synopsys have worked with TSMC and other foundries for decades on final design checks for DRC and LVS compliance.

“the unit conforms to the rules that that foundry has set.”

Claim: Final design checks determine whether a chip design conforms to the relevant foundry's rules.

“But could you have some kind of fuzzy placement algorithm in the meantime that gets you almost all the way to your final helps design,”

Claim: An approximate placement algorithm could bring a chip design close to completion before the final sign-off stage.

“You know, I think that is exactly where people should be doing more work because otherwise that is the bottleneck.”

Claim: People should do more work to iterate faster because otherwise iteration becomes the bottleneck.

“You know, I think that is exactly where people should be doing more work because otherwise that is the bottleneck.”

Claim: More work should be done on approximate placement methods to prevent placement from becoming a bottleneck.

“and I think this is true of, you know, RSI for AI experiments as well.”

Claim: The speaker thinks the same point applies to recursive self-improvement for AI experiments.

“Well, is as we do become, you know, as our intelligence levels on the thinking that then drives that experiment gets higher and higher, and as that experiment therefore becomes the bottleneck, I think we do more thinking. And this is just generally fractals workload bets.”

Claim: As intelligence used to think through experiments increases, the experiments themselves become the bottleneck, prompting more thinking; this is generally one of Fractile's workload bets.

“It's the reason we're so excited about hard problems.”

Claim: Fractile is excited about working on hard problems.

“And it's kind of the reason that we see fractal as the chip that takes us toward accelerating the solution to.”

Claim: Fractile sees its chip as a means of accelerating solutions to hard problems.

“hard problems in general is i think it becomes absolutely it's like almost uh you know a moral compunction to think harder before every single experiment that you fire off um i think uh as maybe baron millage on doakash recently who said you know um that perhaps you should think for 100 years you know before you fire off an experiment and then think for another 100 years of human equivalent with these models about the results of that AI experiment because the experiment in the middle is pretty expensive and takes a certain amount of wall clock time and so I think you get there also with things like chip design where there are these kind of fundamental wall clock time bottlenecks for the more conventional algorithms we're going to be generating a lot of reasoning tokens before we go and place a given circuit.”

Claim: The speaker cites a recent suggestion to think for 100 years before launching an experiment and then for another 100 human-equivalent years about the experiment's results.

“hard problems in general is i think it becomes absolutely it's like almost uh you know a moral compunction to think harder before every single experiment that you fire off um i think uh as maybe baron millage on doakash recently who said you know um that perhaps you should think for 100 years you know before you fire off an experiment and then think for another 100 years of human equivalent with these models”

Claim: The speaker regards thinking harder before every experiment as almost a moral compunction.

“about the results of that AI experiment because the experiment in the middle is pretty expensive and takes a certain amount of wall clock time and so I think you get there also with things like chip design where there are these kind of fundamental wall clock time bottlenecks for the more conventional algorithms we're going to be generating a lot of reasoning tokens before we go and place a given circuit.”

Claim: In chip design, conventional algorithms face fundamental wall-clock bottlenecks, and models will generate many reasoning tokens before placing a circuit.

“about the results of that AI experiment because the experiment in the middle is pretty expensive and takes a certain amount of wall clock time and so I think you get there also with things like chip design where there are these kind of fundamental wall clock time bottlenecks for the more conventional algorithms we're going to be generating a lot of reasoning tokens before we go and place a given circuit.”

Claim: AI experiments are expensive and consume wall-clock time.

“workload bets and then collaborating closely with design partners on the model architectural shifts, hopefully a little ahead so you can actually do some planning.”

Claim: Fractile makes workload bets and works closely with design partners on model-architecture shifts, ideally anticipating them early enough to plan.

“The biggest thing that we're doing, of course, is this kind of speed chasing, Maximizing memory bandwidth in order to be able to run these models faster.”

Claim: Fractile's biggest current focus is increasing speed by maximizing memory bandwidth so models can run faster.

“And one of the dualities I think you have to span as a chip company is to build something that is”

Claim: A chip company must build for a broad range of anticipated workloads.

“that is both exceptionally good at the kind of gamut of workloads that are coming down the pipe under this idea of the kind of hardware lottery where these will be trained on HBM-based GPUs or XPUs.”

Claim: Under the hardware-lottery dynamic, future workloads will be trained on HBM-based GPUs or XPUs.

“And so it's great that this is then the chip that's exceptional at taking kind of today's auto-regressive MOE transformers and running them much faster and more efficiently.”

Claim: Fractile's chip runs today's autoregressive mixture-of-experts transformers faster and more efficiently.

“But I think what's exciting about building chips with kind of fundamentally new capabilities is you then also get to explore whether there are kind of gravitational pulls that you yourself get to pull the new model landscape in because of those properties and so for fractal you know this idea of having like 25 times more bandwidth per chip than a than an hbm-based chip we get to explore for instance internally these kind of ideas of like scaling laws for bandwidth basically you know the scaling laws traditionally we think of them as like flop scaling laws you know we”

Claim: Building chips with fundamentally new capabilities may allow their creators to influence the direction of the model landscape.

“is you then also get to explore whether there are kind of gravitational pulls that you yourself get to pull the new model landscape in because of those properties and so for fractal you know this idea of having like 25 times more bandwidth per chip than a than an hbm-based chip we get to explore for instance internally these kind of ideas of like scaling laws for bandwidth basically you know the scaling laws traditionally we think of them as like flop scaling laws you know we”

Claim: Fractile is exploring whether model performance follows scaling laws for bandwidth.

“is you then also get to explore whether there are kind of gravitational pulls that you yourself get to pull the new model landscape in because of those properties and so for fractal you know this idea of having like 25 times more bandwidth per chip than a than an hbm-based chip we get to explore for instance internally these kind of ideas of like scaling laws for bandwidth basically you know the scaling laws traditionally we think of them as like flop scaling laws you know we”

Claim: Fractile's chip provides about 25 times the bandwidth per chip of an HBM-based chip.

“is you then also get to explore whether there are kind of gravitational pulls that you yourself get to pull the new model landscape in because of those properties and so for fractal you know this idea of having like 25 times more bandwidth per chip than a than an hbm-based chip we get to explore for instance internally these kind of ideas of like scaling laws for bandwidth basically you know the scaling laws traditionally we think of them as like flop scaling laws you know we uh we get better and better performance the more flops we pour into these models at train time and test time um and if you look at just like the known landscape of ideas today we already see some scaling laws for bandwidth so one is uh these mixture of expert models it's pretty well known that actually ideally we would make them sparser and sparser and sparser so for like iso intelligence you will save a ton of flop”

Claim: Traditional scaling laws describe models improving as more FLOPs are allocated at training and test time.

“uh we get better and better performance the more flops we pour into these models at train time and test time um and if you look at just like the known landscape of ideas today we already see some scaling laws for bandwidth so one is uh these mixture of expert models it's pretty well known that actually ideally we would make them sparser and sparser and sparser so for like iso intelligence you will save a ton of flop”

Claim: For a fixed level of intelligence, increasing sparsity in mixture-of-experts models can substantially reduce FLOP requirements.

“uh we get better and better performance the more flops we pour into these models at train time and test time um and if you look at just like the known landscape of ideas today we already see some scaling laws for bandwidth so one is uh these mixture of expert models it's pretty well known that actually ideally we would make them sparser and sparser and sparser so for like iso intelligence you will save a ton of flop”

Claim: For the same intelligence level, increasing MoE sparsity from 1-in-16 to 1-in-128 or 1-in-256 can save substantial FLOPs.

“flops if you go from being like one in 16 sparse on your moe to one in 128 one in 256 um but one of the challenges one of the headwinds to doing that is actually that it becomes incredibly prohibitive on today's hbm-based gpus xpus to serve those models efficiently you end up often bandwidth bottlenecks you end up serving it really really low mfu low flops utilization”

Claim: Today's HBM-based GPUs and XPUs often cannot efficiently serve highly sparse MoE models because of bandwidth bottlenecks, resulting in low FLOP utilization.

“And so there are these areas where even as you build for today's world, you get to kind of look at where you can unlock greater capability.”

Claim: Building for current workloads can reveal opportunities to unlock greater capabilities.

“with attention. There are forms of attention that are less bandwidth hungry. They tend to be more flops hungry for a certain level of intelligence.”

Claim: Some attention mechanisms are less bandwidth-intensive but more FLOP-intensive at a given intelligence level.

“this property of memory bandwidth which hasn't really been scaled so much on chips recently we've scaled flops like a million fold in the last 20 years memory bandwidth has gone up about 40x in the same time frame so as you scale that frontier you get to conserve more of the other thing we get to reduce the number of flops that we're using on these models to get to a certain level of intelligence and so that's the kind of really a multiplying factor on you”

Claim: Over the last 20 years, chip FLOPs increased about one million-fold, compared with about a 40-fold increase in memory bandwidth.

“this property of memory bandwidth which hasn't really been scaled so much on chips recently we've scaled flops like a million fold in the last 20 years memory bandwidth has gone up about 40x in the same time frame so as you scale that frontier you get to conserve more of the other thing we get to reduce the number of flops that we're using on these models to get to a certain level of intelligence and so that's the kind of really a multiplying factor on you global throughput for these models.”

Claim: Increasing memory bandwidth can act as a multiplier for models' global throughput.

“this property of memory bandwidth which hasn't really been scaled so much on chips recently we've scaled flops like a million fold in the last 20 years memory bandwidth has gone up about 40x in the same time frame so as you scale that frontier you get to conserve more of the other thing we get to reduce the number of flops that we're using on these models to get to a certain level of intelligence and so that's the kind of really a multiplying factor on you”

Claim: Chip memory bandwidth has scaled much less than FLOPs in recent years.

“this property of memory bandwidth which hasn't really been scaled so much on chips recently we've scaled flops like a million fold in the last 20 years memory bandwidth has gone up about 40x in the same time frame so as you scale that frontier you get to conserve more of the other thing we get to reduce the number of flops that we're using on these models to get to a certain level of intelligence and so that's the kind of really a multiplying factor on you global throughput for these models.”

Claim: Increasing memory bandwidth can reduce the FLOPs required for models to achieve a given level of intelligence.

“you know behaviour where really everybody who is deploying at scale is kind of trying to deploy as many separate platforms as they can I think one thing that will remain the case is that there will be a real need for kind of a diversity of supply as something like compute becomes so existential to these players there's a bit of a joke today that the sort of first party efforts their primary purpose is to reduce the price that people pay Nvidia.”

Claim: There is a joke that first-party chip efforts primarily aim to reduce buyers' payments to Nvidia.

“you know behaviour where really everybody who is deploying at scale is kind of trying to deploy as many separate platforms as they can I think one thing that will remain the case is that there will be a real need for kind of a diversity of supply as something like compute becomes so existential to these players there's a bit of a joke today that the sort of first party efforts their primary purpose”

Claim: Large-scale AI deployers are currently trying to use as many separate compute platforms as possible.

“you know behaviour where really everybody who is deploying at scale is kind of trying to deploy as many separate platforms as they can I think one thing that will remain the case is that there will be a real need for kind of a diversity of supply as something like compute becomes so existential to these players there's a bit of a joke today that the sort of first party efforts their primary purpose”

Claim: Large AI players will continue to need diverse compute supply because compute is existential to them.

“is to reduce the price that people pay Nvidia. And there may be some truth to that as well, because those efforts are kind of architecturally quite similar, right? So it's a bet that is not on enabling a fundamental capability that those other chips cannot enable.”

Claim: Some alternative-chip efforts may aim to reduce the price customers pay Nvidia; these efforts are architecturally similar to Nvidia’s chips and are not bets on capabilities Nvidia’s chips cannot enable.

“And there may be some truth to that as well, because those efforts are kind of architecturally quite similar, right? So it's a bet that is not on enabling a fundamental capability that those other chips cannot enable.”

Claim: First-party chip efforts are not bets on enabling fundamental capabilities unavailable from other chips.

“And there may be some truth to that as well, because those efforts are kind of architecturally quite similar, right? So it's a bet that is not on enabling a fundamental capability that those other chips cannot enable.”

Claim: The speaker thinks there may be some truth to the idea that first-party chip efforts aim to lower Nvidia prices, since those efforts are architecturally quite similar.

“I think it is this kind of gamesmanship of, you know, can we build some of this over here, or can I buy something from AMD so that maybe I then get a better price from Nvidia and so on, and a play also for kind of aggregate capacity and a bit of control.”

Claim: AI labs may build chips or buy AMD chips to gain leverage for better Nvidia pricing, aggregate compute capacity, and some control.

“I think it is this kind of gamesmanship of, you know, can we build some of this over here, or can I buy something from AMD so that maybe I then get a better price from Nvidia and so on, and a play also for kind of aggregate capacity and a bit of control.”

Claim: Large AI players may build compute internally or buy from AMD to negotiate better Nvidia pricing, while seeking aggregate capacity and some control.

“I think as you then see this move into chips that actually give new capabilities, you know, that's a place where, you know, I suddenly see for what we're doing a real need for everybody that wants to deploy AI at the frontier, they have to have some solution to running these models orders of magnitude faster.”

Claim: The speaker believes that as chips provide new capabilities, every frontier AI deployer will need a solution capable of running models orders of magnitude faster.

“there's this window of kind of premium capability, premium intelligence that is their entire raison d'etre.”

Claim: Frontier labs’ purpose centers on maintaining a window of premium capability and intelligence.

“batch you know otherwise we'd all be using chemi models all the time and so as you do have this pressure from behind from open source you know it does become clearly even more important for folks that want to deploy at the frontier to have every aspect of what that means so the best model waits but also then the fastest deployment so you can do the most reasoning in the in the shortest amount of time and so I think this kind of premium end of the chip space where you're up you're optimizing for speed over everything else becomes a kind of very important part of this part of this world.”

Claim: The speaker considers the premium chip segment, which prioritizes speed above all else, an important part of the AI landscape.

“batch you know otherwise we'd all be using chemi models all the time and so as you do have this pressure from behind from open source you know it does become clearly even more important for folks that want to deploy at the frontier to have every aspect of what that means so the best model waits but also then the fastest deployment so you can do the most reasoning in the in the shortest amount of time and so I think this kind of premium end of the chip space where you're up”

Claim: Because of pressure from open source, frontier AI deployers need both the best model and the fastest deployment to maximize reasoning performed in the shortest time.

“you know, if you look at Fractal, we try to be as aligned as possible with the needs of the frontier. We try to look a bit like some of those teams as well. We have people that really think deeply about these workloads and so on.”

Claim: Fractile says it aligns its work with frontier labs’ needs and employs people who study frontier workloads deeply.

“make is explaining why we believe that over you know multiple decades the world sustains some kind of frontier third party chip players you know why does this not just roll up into these into these labs um and i think that comes down to the kind of asymmetric game that all of these labs and frontier players have to play uh where they're exposed to an enormous risk if they go all in on a hardware”

Claim: The speaker believes third-party frontier chip companies can persist for multiple decades because frontier labs face asymmetric risks from going all in on hardware.

“Suppose I'm, you know, I'm lab one, and I've gone all in on some proprietary silicon, and then lab two discovers some new computational breakthrough that delivers far better computational efficiencies for the same level of intelligence, but it only works on the chip that they've decided to deploy. I could die in the nine months before I get to deploy enough of that chip that I've also now gained that kind of 5x in computational efficiency. And so there is a really”

Claim: A lab that commits fully to proprietary silicon could lose out if a rival discovers a more computationally efficient method at the same intelligence level that only runs on the rival’s chip, because the first lab may not deploy enough of its own chip in time.

“urgent need for these chips to actually deploy the same these companies to deploy the same platforms as one another they're playing different games right they're trying to now compete you know in the model where um i think having deeply differentiated bets and going all in on those differentiated bets at the chip player is sort of an irrational and a very dangerous move for anybody that is playing at the frontier thanks”

Claim: The speaker argues that frontier companies urgently need to deploy compatible platforms, and that chip companies serving the frontier should not make and fully commit to deeply differentiated bets because doing so is irrational and dangerous.