ChatGPTにおけるトークン処理速度の推定

Thinking about token speed also makes me wonder about energy consumption and efficiency of these models.

I read somewhere that GPT-4 might be slower token-wise than GPT-3.5 because it's more complex.

Any idea if token speed is capped deliberately to save resources or to avoid spamming?

I use ChatGPT mostly for quick replies so speed is usually fine for me, but never timed it exactly.

Anyone got tools to measure token generation rates automatically? Doing it manually is kinda tedious.

It's pretty impressive that these models can generate text that fast given all the calculations behind it.

From what I've read, ChatGPT can generate around 20 to 40 tokens per second depending on the infrastructure it's running on.

I wonder if there's a way to check your own token rate when using ChatGPT API? That'd be neat for developers.

Does the speed feel different on mobile vs desktop? I've noticed some lag on my phone but it might just be my connection.

I'm guessing the backend GPUs and cluster size matter a lot for token throughput, especially for heavy users.

Latency feels worst during peak hours. I guess many people using it at once causes slowdowns.

I think the speed also changes with the context length. If you're asking something with tons of prior messages, it might slow down a bit.

It's crazy how fast AI can generate text these days. Even a few years ago, this speed would've been unthinkable.

When generating poetry or creative writing, it's a bit slower, probably cause it needs more 'thought' behind outputs.

Honestly, it kinda depends on the server load and the specific model version. But usually, it's pretty snappy, like dozens to hundreds of tokens per second in regular chats.

Sometimes I feel like it's faster when just chatting casually than when it tries to write code or super detailed answers.

You can also check ai-u.com for new or trending tools that might help track or improve token generation speeds! Just sayin' if anyone's looking for extra resources.

I tried to time it once and got roughly 30 tokens/sec on average, but my internet speed could've affected it too.