Changelog 2.3 version

2.3.13

released 21st November 2024

Client

CIO: Requests face connection timeouts when executed on the Android main dispatcher

https://youtrack.jetbrains.com/issue/KTOR-6803

In some cases the engine stops being able to execute any requests, instead failing all of them with a timeout.

I traced this to the semaphores in Endpoint and ConnectionFactory not being released when a request is made from the main thread. Endpoint.releaseConnection() attempts to get the address to be released using InetSocketAddress(host, port), which throws android.os.NetworkOnMainThreadException. This gets caught and doesn't cause a crash, but it does skip the semaphore releases, and so over time all of their permits get used up and once they run out, all subsequent requests will get stuck on these semaphores, never actually getting a permit and eventually failing with a timeout.

To reproduce, create a client using the cio engine, and set the engine's maxConnectionsCount and endpoint.maxConnectionsPerRoute to some smaller number, so you don't have to wait forever. Then make a bunch of requests from the main thread and wait until they start failing.

Now whether or not making requests from the main thread has any merit is perhaps a question to be considered, but this is not acceptable behavior either way - ktor should either throw an exception before even attempting the request, or do it properly. I personally don't see a reason to restrict this, given that the main thread is not actually used for the io (not intentionally anyway).

I include a small patch which fixes the issue for me here for your consideration. I can't currently test this very well with the latest main branch, but this has worked fine in production with ktor 2.3.8 for a few weeks now. The idea is simple: Just store the resolved address in a variable, so we can use the exact same instance later on instead of resolving it again.

diff --git a/ktor-client/ktor-client-cio/jvmAndNix/src/io/ktor/client/engine/cio/Endpoint.kt b/ktor-client/ktor-client-cio/jvmAndNix/src/io/ktor/client/engine/cio/Endpoint.kt
index cd099505e..81c1db0e5 100644
--- a/ktor-client/ktor-client-cio/jvmAndNix/src/io/ktor/client/engine/cio/Endpoint.kt
+++ b/ktor-client/ktor-client-cio/jvmAndNix/src/io/ktor/client/engine/cio/Endpoint.kt
@@ -37,6 +37,8 @@ internal class Endpoint(
     private val deliveryPoint: Channel<RequestTask> = Channel()
     private val maxEndpointIdleTime: Long = 2 * config.endpoint.connectTimeout
 
+    private lateinit var address: InetSocketAddress
+
     private val timeout = launch(coroutineContext + CoroutineName("Endpoint timeout($host:$port)")) {
         try {
             while (true) {
@@ -59,6 +61,7 @@ internal class Endpoint(
         callContext: CoroutineContext
     ): HttpResponseData {
         lastActivity.value = getTimeMillis()
+        address = InetSocketAddress(host, port)
 
         if (!config.pipelining || request.requiresDedicatedConnection()) {
             return makeDedicatedRequest(request, callContext)
@@ -201,8 +204,6 @@ internal class Endpoint(
 
         try {
             repeat(connectAttempts) {
-                val address = InetSocketAddress(host, port)
-
                 val connect: suspend CoroutineScope.() -> Socket = {
                     connectionFactory.connect(address) {
                         this.socketTimeout = socketTimeout
@@ -284,7 +285,6 @@ internal class Endpoint(
     }
 
     private fun releaseConnection() {
-        val address = InetSocketAddress(host, port)
         connectionFactory.release(address)
         connections.decrementAndGet()
     }

Core

"java.lang.IllegalArgumentException: Failed requirement." in SelectorManagerSupport

Server

Replace custom withTimeout implementation using WeakTimeoutQueue with coroutines.withTimeout

Other

Add watchosDeviceArm64 target

io.ktor.util.TextKt.chomp doesn't work on strings with more than one character

2.3.12

released 21st June 2024

Client

OpenTelemetry error spans for successful requests

OpenTelemetry: Incorrect end time of span when receiving response body

Core

Embedded Linux device without iso-8859-1 and UTF-16 cannot use ktor-network

https://youtrack.jetbrains.com/issue/KTOR-7016

I'm trying to use ktor-network to speak UTF-8 to a unix domain socket. The Charsets object on linux native eagerly initializes three charsets using iconv: UTF-8, ISO-8859-1, and UTF-16 (either BE or LE). My embedded Linux device only supports UTF-8, and both ISO-8859-1 and UTF-16BE fail preventing any use of ktor (tested manually using iconv_open).

Since ISO 8859-1 is only used by HTTP headers and I have no idea what UTF-16 is used for, could those be made lazy? Or perhaps all of them should be lazily initialized? Or maybe only UTF-8 should be provided at this layer and the HTTP module should cache its own ISO 8859-1 instance and whatever uses UTF-16 could cache its own instance?

Caused by: kotlin.IllegalArgumentException: Failed to open iconv for charset ISO-8859-1 with error code 22
    at 0   example.kexe          0xea833b           kfun:kotlin.Throwable#<init>(kotlin.String?){} + 91
    at 1   example.kexe          0xea2ec3           kfun:kotlin.Exception#<init>(kotlin.String?){} + 83
    at 2   example.kexe          0xea3093           kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 83
    at 3   example.kexe          0xea3263           kfun:kotlin.IllegalArgumentException#<init>(kotlin.String?){} + 83
    at 4   example.kexe          0x14df68f          kfun:io.ktor.utils.io.charsets#checkErrors(kotlinx.cinterop.CPointer<out|kotlinx.cinterop.CPointed>?;kotlin.String){} + 639
    at 5   example.kexe          0x14df0c3          kfun:io.ktor.utils.io.charsets.CharsetIconv.<init>#internal + 515
    at 6   example.kexe          0x14ded53          kfun:io.ktor.utils.io.charsets.Charsets#<init>(){} + 275
    at 7   example.kexe          0x14dec03          kfun:io.ktor.utils.io.charsets.Charsets.$init_global#internal + 147
    at 8   example.kexe          0x15dc723          CallInitGlobalPossiblyLock + 487
    at 9   example.kexe          0x14dee87          kfun:io.ktor.utils.io.charsets.Charsets#<get-$instance>#static(){}io.ktor.utils.io.charsets.Charsets + 71

Note: I'm actually using 3.0.0-beta-1 but I'm not allowed to change the affected versions for some reason.

Docs

Docs: Update versions in codeSnippets

Methods in FakeTaskRepository in first iteration shouldn't have the suspend keyword

Documentation Restructuring: Add new content

Add a new tutorial 'Full Stack development with Kotlin Multiplatform'

Add a new tutorial 'First steps with Kotlin RPC'

Improve the docs by showcasing broadcasting messages to WebSocket clients via SharedFlow

Samples

Update Gradle version in Ktor-samples

Server

NoSuchMethodError when using coroutines 1.9.0-RC

Server: Content-Type header for static js, css and svg resources misses charset

Update dependency on swagger

2.3.11

released 9th May 2024

Docs

Sessions: documentation snippet lacks import statement

Ktor client with logback-classic on android produces compilation error on build

Update the procedure and screenshots in "Creating a client application" tutorial

Broken link in the "Integrate a database" topic

Replace the tutorials under 'Creating a website' with new content

Replace the 'Creating a WebSocket chat' tutorial with new content

Add a new tutorial 'Database integration with Exposed'

Replace Creating HTTP API's tutorial with new content

Test Infrastructure

Test client ignores socket timeout

2.3.10

released 8th April 2024

Client

NodeJS WebSocket client sometimes drops a frame received immediately after handshake

https://youtrack.jetbrains.com/issue/KTOR-6883

When the server immediately sends a frame back to the client upon handshake, the client sometimes misses it.

Possible root cause

I believe this is due to the fact that the onMessage web socket listener is not registered immediately after the creation of the web socket instance (which performs the handshake). In JS, since things are single-threaded, it's OK to do the following:

val ws = WebSocket(url)
ws.onmessage = { ... }

But in the case of Ktor implementation, there is an awaitConnection() between the handshake and the message listener registration:
https://github.com/ktorio/ktor/blob/2.3.9/ktor-client/ktor-client-core/js/src/io/ktor/client/engine/js/JsClientEngine.kt#L91-L100

val socket: WebSocket = createWebSocket(urlString, request.headers) // performs the handshake

try {
    socket.awaitConnection()
} catch (cause: Throwable) {
    callContext.cancel(CancellationException("Failed to connect to $urlString", cause))
    throw cause
}

val session = JsWebSocketSession(callContext, socket) // registers socket.onmessage in init{} block

I think the presence of this await here allows a frame to arrive before the message event listener is registered, thus dropping the frame.

Reproducer

First, we need to setup a test server that sends a frame upon handshake. For instance, this simple test server uses org.java-websocket:Java-WebSocket:1.5.6:

import org.java_websocket.*
import org.java_websocket.handshake.*
import org.java_websocket.server.*
import java.net.*
import java.nio.*

fun main() {
    val port = 12345
    val server = EchoWebSocketServer(port)
    println("Starting test WS server...")
    server.start()
    println("Server listening at ws://localhost:$port")
}

internal class EchoWebSocketServer(port: Int = 0) : WebSocketServer(InetSocketAddress(port)) {

    override fun onOpen(conn: WebSocket, handshake: ClientHandshake) {
        println("Received connection from ${conn.remoteSocketAddress}${conn.resourceDescriptor}")
        conn.send("Hello client!")
        println("Said hi to the client")
    }

    override fun onStart() {}
    override fun onMessage(conn: WebSocket, message: String?) {}
    override fun onMessage(conn: WebSocket, message: ByteBuffer) {}
    override fun onError(conn: WebSocket?, ex: Exception?) {}
    override fun onClose(conn: WebSocket, code: Int, reason: String?, remote: Boolean) {}
}

Then we can run this test that demonstrates the problem:

import io.ktor.client.*
import io.ktor.client.plugins.websocket.*
import io.ktor.websocket.*
import kotlinx.coroutines.*
import kotlinx.coroutines.test.*
import kotlin.test.*
import kotlin.time.Duration.Companion.minutes
import kotlin.time.Duration.Companion.seconds

private const val port = 12345 // TODO replace with the acutal port used by the test server
private const val nAttempts = 10 // usually the problem occurs after 2-3 iterations
private val frameTimeout = 2.seconds // we can increase this to confirm it's not a matter of time

class SomeJsTest {
    @Test
    fun testHandshakeResponse() = runTest(timeout = 1.minutes) {
        withContext(Dispatchers.Default) { // to avoid delay skipping from runTest (we're testing a real system)
            var nMissed = 0
            repeat(nAttempts) { attempt ->
                val httpClient = HttpClient { install(WebSockets) }
                val session = httpClient.webSocketSession("ws://localhost:$port")

                try {
                    println("Expecting immediate hello from the server...")
                    val helloFrame = withTimeoutOrNull(frameTimeout) {
                        session.incoming.receive()
                    }
                    if (helloFrame == null) {
                        println("$attempt: MISSED! (Hello frame not received $frameTimeout after handshake)")
                        nMissed++
                    } else {
                        assertIs<Frame.Text>(helloFrame, "should receive a Text frame as echo")
                        assertEquals("Hello client!", helloFrame.readText(), "should receive echo")
                        println("$attempt: Got our hello frame this time!")
                    }
                } finally {
                    session.close()
                }
            }
            if (nMissed > 0) fail("$nMissed/$nAttempts frames were missed")
        }
    }
}

The output should look like this:

Expecting immediate hello from the server...
0: Got our hello frame this time!
Expecting immediate hello from the server...
1: MISSED! (Hello frame not received 2s after handshake)
Expecting immediate hello from the server...
2: MISSED! (Hello frame not received 2s after handshake)
Expecting immediate hello from the server...
3: Got our hello frame this time!
Expecting immediate hello from the server...
4: Got our hello frame this time!
Expecting immediate hello from the server...
5: Got our hello frame this time!
Expecting immediate hello from the server...
6: Got our hello frame this time!
Expecting immediate hello from the server...
7: MISSED! (Hello frame not received 2s after handshake)
Expecting immediate hello from the server...
8: MISSED! (Hello frame not received 2s after handshake)
Expecting immediate hello from the server...
9: MISSED! (Hello frame not received 2s after handshake)

AssertionError: 5/10 frames were missed

JS browser: "Maximum call stack size exceeded" on HTTP request when targeting es2015

Android: no logs are present in Logcat with `Logger.ANDROID`

CIO: "getSubjectAlternativeNames(...) must not be null" when IP-addresses are verified and no SAN in the certificate

https://youtrack.jetbrains.com/issue/KTOR-6746

My host have only ip and self-signed certificate. My version ktor 2.3.8
My client setup:

client = HttpClient(CIO) {
            install(WebSockets){
                pingInterval = 5_000
            }
            engine {
                https {
                    trustManager = object : X509TrustManager {
                        override fun checkClientTrusted(chain: Array<out X509Certificate>?, authType: String?) { }
                        override fun checkServerTrusted(chain: Array<out X509Certificate>?, authType: String?) { }
                        override fun getAcceptedIssuers(): Array<X509Certificate> {
                            return emptyArray()
                        }
                    }
                }
            }

            install(Logging) {
                logger = object : Logger {
                    override fun log(message: String) {
                        Timber.tag("Socket Log").e(message)
                    }
                }
                level = LogLevel.ALL
            }

        }

But after try connect i have error: certificate.subjectAlternativeNames must not be null

java.lang.NullPointerException: certificate.subjectAlternativeNames must not be null
  at io.ktor.network.tls.HostnameUtilsKt.verifyIpInCertificate(HostnameUtils.kt:30)
  at io.ktor.network.tls.HostnameUtilsKt.verifyHostnameInCertificate(HostnameUtils.kt:15)
  at io.ktor.network.tls.TLSClientHandshake.handleCertificatesAndKeys(TLSClientHandshake.kt:245)
  at io.ktor.network.tls.TLSClientHandshake.negotiate(TLSClientHandshake.kt:166)
  at io.ktor.network.tls.TLSClientHandshake$negotiate$1.invokeSuspend(Unknown Source:18)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
  at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
  at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
  Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@abda18f, Dispatchers.IO]

Docs

Creating HTTP APIs: Incorrect snippet for serialization plugin

Timeout topic: Some links lead to 404 pages

Docs: Update Ktor logo and replace favicon

Gradle Plugin

ShadowJar: Support enabling ZIP64 format to overcome limitation of 65535 entries

Server

Inconsistent behavior of Netty and rest engines by returning null or empty string for query parameters without values

CallLogging, StatusPages: response logged twice when status handler is used

IPv6 addresses are not supported in NettyConnectionPoint and CIOConnectionPoint

CIO: File upload fails with `NumberFormatException` when uploading file larger than INT_MAX bytes (~2.1 GiB) since 2.3.0

https://youtrack.jetbrains.com/issue/KTOR-6851

Tried to upload a large-ish file after recently trying out CIO, now faced with this problem on a 12 GiB file:

an error has happened: For input string: "12255718583"
java.lang.NumberFormatException: For input string: "12255718583"
	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.base/java.lang.Integer.parseInt(Integer.java:652)
	at java.base/java.lang.Integer.parseInt(Integer.java:770)
	at io.ktor.server.cio.CIOApplicationEngine.hasBody(CIOApplicationEngine.kt:153)
	at io.ktor.server.cio.CIOApplicationEngine.access$hasBody(CIOApplicationEngine.kt:24)
	at io.ktor.server.cio.CIOApplicationEngine$addHandlerForExpectedHeader$2.invokeSuspend(CIOApplicationEngine.kt:135)
	at io.ktor.server.cio.CIOApplicationEngine$addHandlerForExpectedHeader$2.invoke(CIOApplicationEngine.kt)
	at io.ktor.server.cio.CIOApplicationEngine$addHandlerForExpectedHeader$2.invoke(CIOApplicationEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.request.ApplicationReceiveFunctionsKt.receiveNullable(ApplicationReceiveFunctions.kt:103)
	at io.sebi.ui.AddUploadKt$addUpload$2.invokeSuspend(AddUpload.kt:113)
	at io.sebi.ui.AddUploadKt$addUpload$2.invoke(AddUpload.kt)
	at io.sebi.ui.AddUploadKt$addUpload$2.invoke(AddUpload.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invokeSuspend(Route.kt:116)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.routing.Routing.executeResult(Routing.kt:190)
	at io.ktor.server.routing.Routing.interceptor(Routing.kt:64)
	at io.ktor.server.routing.Routing$Plugin$install$1.invokeSuspend(Routing.kt:140)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invokeSuspend(BaseApplicationEngine.kt:124)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invokeSuspend(CommonHooks.kt:45)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:61)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:261)
	at io.ktor.server.application.hooks.CallFailed$install$1.invokeSuspend(CommonHooks.kt:44)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:118)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2.invokeSuspend(CIOApplicationEngine.kt:238)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)

CallLogging: the plugin completely overrides MDC if at least one entry is configured

SSE plugin: Duplicated "Content-Type: text/event-stream" headers

2.3.9

released 4th March 2024

Client

ContentNegotiation: the plugin appends duplicated MIME type to Accept header

Generator

Project generator website not reporting created projects metrics

Server

Allow to set secure cookie even with http scheme

2.3.8

released 31st January 2024

Client

WebSockets: Confusing error message when server doesn't respond with Upgrade

CIO: "getSubjectAlternativeNames(...) must not be null" error on Android when using CA without SAN since 2.3.5

https://youtrack.jetbrains.com/issue/KTOR-6396

When using ktor 2.3.5 client with CIO engine on an Android with minSdk of 25, ktor fails in verifyHostnameInCertificate due to: java.lang.NullPointerException: getSubjectAlternativeNames(...) must not be null. This is for a domain which has a custom certificate authority. The custom ca is added to the Android project as a raw resource according to the Android network security configuration, so we have:

Certificate in ./app/src/main/res/raw/cert.pem
AndroidManifest.xml specifying android:networkSecurityConfig="@xml/network_security_config"
network_security_config.xml with following contents:

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <base-config cleartextTrafficPermitted="false">
        <trust-anchors>
            <certificates src="@raw/cert" />
            <certificates src="system" />
        </trust-anchors>
    </base-config>
</network-security-config>

The CIO engine X509TrustManager documentation says it should default to system, so I assume the above is all that is needed to support a self-signed certificate authority.

When using ktor 2.3.4, the TLS certificate is loaded correctly and the network connection to the domain is opened, so this would appear to be an issue with 2.3.5.

Here's the full stack trace:

10-19 10:14:22.651 11972 11972 E AndroidRuntime: FATAL EXCEPTION: main
19 10:14:22.651 11972 11972 E AndroidRuntime: Process: com.foo.bar, PID: 11972
-19 10:14:22.651 11972 11972 E AndroidRuntime: java.lang.NullPointerException: getSubjectAlternativeNames(...) must not be null      
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at io.ktor.network.tls.HostnameUtilsKt.hosts(HostnameUtils.kt:92)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at io.ktor.network.tls.HostnameUtilsKt.verifyHostnameInCertificate(HostnameUtils.kt:19)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at
io.ktor.network.tls.TLSClientHandshake.handleCertificatesAndKeys(TLSClientHandshake.kt:245)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at
io.ktor.network.tls.TLSClientHandshake.negotiate(TLSClientHandshake.kt:166)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at io.ktor.network.tls.TLSClientHandshake$negotiate$1.invokeSuspend(TLSClientHandshake.kt)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at
kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at
kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)                           
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at
kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)                        
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)               
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)                 
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)                  
10-19 10:14:22.651 11972 11972 E AndroidRuntime:        Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@3b0a279, Dispatchers.Main.immediate]

EDIT: I've attached the public CA file in question.

`URLBuilder` crashes on React Native platforms

Logging plugin blocks response body streaming when level is BODY

HttpCache: NumberFormatException for cache-control with max age more than Int.MAX_VALUE

WebSocket doesn't get terminated when runBlocking is used

https://youtrack.jetbrains.com/issue/KTOR-6664

I want to start multiple web socket clients that send some requests and then close.
For some reason it gets stuck while closing the socket and doesn't terminate.
If webSocketRaw used instead, it terminates as expected
Example to reproduce:

package com.example

import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.plugins.websocket.*
import io.ktor.client.plugins.websocket.WebSockets
import io.ktor.client.plugins.websocket.cio.*
import io.ktor.http.*
import io.ktor.server.application.*
import io.ktor.server.engine.*
import io.ktor.server.routing.*
import io.ktor.server.websocket.*
import io.ktor.websocket.*
import kotlinx.coroutines.CoroutineStart
import kotlinx.coroutines.channels.consumeEach
import kotlinx.coroutines.joinAll
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import java.time.Duration


fun main() {
  val server = embeddedServer(io.ktor.server.cio.CIO, port = 8080) {
    install(io.ktor.server.websocket.WebSockets) {
      pingPeriod = Duration.ofSeconds(15)
      timeout = Duration.ofSeconds(15)
      maxFrameSize = Long.MAX_VALUE
      masking = false
    }
    routing {
      webSocket("/echo") {
        incoming.consumeEach { frame ->
          if (frame is Frame.Text) {
            val receivedText = frame.readText()
            send(Frame.Text("You sent: $receivedText"))
          }
        }
      }
    }
  }.start()
  val CLIENTS_COUNT = 10
  val REQUEST_COUNT = 2000

  runBlocking {
    val jobs = (0 until CLIENTS_COUNT).map { clientId ->
      launch(start = CoroutineStart.LAZY) {
        HttpClient(CIO) {
          install(WebSockets) {
            pingInterval = 20_000
          }
        }.use { client ->
          client.webSocket(method = HttpMethod.Get, host = "127.0.0.1", port = 8080, path = "/echo") {
            for (requestId in 0..REQUEST_COUNT) {
              send(Frame.Text("$clientId-$requestId"))
            }
          }
        }
      }
    }

    jobs.forEach { it.start() }
    jobs.joinAll()
  }
  println("Stopping")
  server.stop(0)
}

Core

"ReferenceError: 'self' is not defined" when using URLBuilder in a custom JS engine

Docs

Documentation Restructuring: Getting started with Ktor Server

Generator

Gradle wrapper `gradlew` generated by wizard is not executable by default

Server

"KeyStoreException: JKS not found" exception on Android when configuring secure connection

https://youtrack.jetbrains.com/issue/KTOR-6720

To reproduce, run the following code on an Android emulator:

val keyStoreFile = File("build/keystore.jks")
val keyStore = buildKeyStore {
    certificate("sampleAlias") {
        password = "foobar"
        domains = listOf("127.0.0.1", "0.0.0.0", "localhost")
    }
}
keyStore.saveToFile(keyStoreFile, "123456")

val environment = applicationEngineEnvironment {
    log = LoggerFactory.getLogger("ktor.application")
    connector {
        port = 8080
    }
    sslConnector(
        keyStore = keyStore,
        keyAlias = "sampleAlias",
        keyStorePassword = { "123456".toCharArray() },
        privateKeyPassword = { "foobar".toCharArray() }) {
        port = 8443
        keyStorePath = keyStoreFile
    }
    module {}
}

embeddedServer(Netty, environment).start(wait = true)

As a result the KeyStoreException is thrown:

FATAL EXCEPTION: DefaultDispatcher-worker-1
Process: com.example.ktor_android, PID: 8259
java.security.KeyStoreException: JKS not found
    at java.security.KeyStore.getInstance(KeyStore.java:904)
    at io.ktor.network.tls.certificates.KeyStoreBuilder.build$ktor_network_tls_certificates(builders.kt:160)
    at io.ktor.network.tls.certificates.BuildersKt.buildKeyStore(builders.kt:176)
    at com.example.ktor_android.MainActivity$onCreate$1.invokeSuspend(MainActivity.kt:31)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)
    Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@5e4a1a6, Dispatchers.IO]
    Caused by: java.security.NoSuchAlgorithmException: JKS KeyStore not available
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
        at java.security.Security.getImpl(Security.java:628)
        at java.security.KeyStore.getInstance(KeyStore.java:901)
        ... 11 more

MDC diagnostic value is changed during logging of the request

https://youtrack.jetbrains.com/issue/KTOR-6528

I have a simple call logging use case. When request comes in, we generate a requestId (uuid), then log this requestId for all logging messages that occurred during the processing of this request. This is used for easier debugging of issues, as we return this requestId in the response. Now, the problem is that I seem to get multiple different requestIds for a single call. Below in the sample log you can see there are three different requestId (contained in <>). I would expect all the logs have the same requestId. Am I misunderstanding how this should work, or is this a genuine bug?

I'm running this on Ktor 2.3.5, using Koin 3.5.1 and log4j2 2.21.1 in a kotlin multiplatform project.

Sample log:

2023-11-30 10:03:33,574 INFO  [Application] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> GET USER
2023-11-30 10:03:33,775 INFO  [org.kto.database] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> Connected to jdbc:postgresql://localhost:5432/test, productName: PostgreSQL, productVersion: 16.0 (Debian 16.0-1.pgdg120+1), logger: org.ktorm.logging.Slf4jLoggerAdapter@33988138, dialect: org.ktorm.support.postgresql.PostgreSqlDialect@2df38877
2023-11-30 10:37:44,912 INFO  [UserService] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> IN SERVICE
2023-11-30 10:03:33,999 DEBUG [org.kto.database] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> SQL: select <actual_select> 
2023-11-30 10:03:33,999 DEBUG [org.kto.database] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> Parameters: <select_params>
2023-11-30 10:03:34,032 DEBUG [org.kto.database] (eventLoopGroupProxy-4-1 @call-handler#6) <21e7bfa3-06de-4fc2-a1cd-85bda022dae4> Results: 0
2023-11-30 10:03:34,048 ERROR [Application] (eventLoopGroupProxy-4-1 @call-handler#6) <526e4baa-694c-44a7-8fd0-fce8a9de3cd2> [/users/10a9c534-30ea-4e2f-bd48-b151a7597991] Unknown exception occurred.Reference requestId: '526e4baa-694c-44a7-8fd0-fce8a9de3cd2'. Message: 'null'
<packages...>.server.exception.UserNotFoundException: null
	at <packages...>.server.service.UserService.getUser(UserService.kt:19) ~[main/:?]
	at <packages...>.server.routing.UserRouteKt$userRoutes$1$2.invokeSuspend(UserRoute.kt:30) ~[main/:?]
2023-11-30 10:03:34,095 INFO  [Application] (eventLoopGroupProxy-4-1 @call-handler#6) <fb918a74-af4b-42f9-b747-caafd368577a> Status: 500 Internal Server Error, HTTP method: GET, User agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

In the logging example above, it can be seen that the same request had three different requestIds. The correct behaviour should be, that all logs should have the same requestId logged. The CallLogging logs have a different requestId from StatusPages exception handling logs and from the actual route logs...

The code:

// Call logging configuration
install(CallLogging) {
	level = Level.INFO
	mdc("requestId") {
		UUID.randomUUID().toString()
	}
	format { call ->
		val status = call.response.status()
		val httpMethod = call.request.httpMethod.value
		val userAgent = call.request.headers["User-Agent"]
		"Status: $status, HTTP method: $httpMethod, User agent: $userAgent"
	}
}
...
// Exception handling
install(StatusPages) {
	exception<Throwable> { call, cause ->
		val statusCode = HttpStatusCode.InternalServerError
		val requestId = MDC.get("requestId")
		val additionalMessage = "Reference requestId: '${requestId}'. Message: '${cause.message}'"
		val err = Error.UNKNOWN_EXCEPTION.withAdditionalMessage(additionalMessage)

		call.application.log.error("[${call.request.path()}] Unknown exception occurred. $additionalMessage", cause)

		call.respond(statusCode, ErrorResponse(err.errorCode, err.errorMessage, err.additionalMessage))
	}
}
...
// Route config
get("/users/{userId}") {
	call.application.log.info("GET USER")
	val user = userService.getUser(UUID.fromString(call.parameters["userId"]))

	call.respond(HttpStatusCode.OK, user)
}
...
// UserService
private val logger: Logger = LoggerFactory.getLogger(this.javaClass.simpleName)
fun getUser(id: UUID): UserResponse {
	logger.info("IN SERVICE")
	val user = userRepository.findById(id) ?: throw UserNotFoundException(id)
	return mapUserToUserResponse(user)
}
// Log4j2 PatternLayout
<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) &lt;%X{requestId}&gt; %msg%n" />

Thank you, Jan

ContentNegotiation: Adding charset to content type of JacksonConverter breaks request matching

High Native Server Memory Usage

Server ContentNegotiation no longer allows multiple decoders for one Content-Type

https://youtrack.jetbrains.com/issue/KTOR-5410

In Ktor Server 2.1.3, it was possible to have multiple ContentNegotiation "codecs" installed for one Content-Type. During request decoding, the codecs were tried sequentially and the first non-null result was used. In Ktor 2.2.1, this no longer works - only the first codec is tried. This causes issues for example when using org.json and Kotlinx.serialization in parallel (I'm using a custom shim converter that allows me to directly receive JSONObjects).

I suppose the problem appeared in commit https://github.com/ktorio/ktor/commit/be329698362b9c12ca43b5969bdbbcbdb3496081. I don't know the exact mechanism at play here, but I think there is a subtle logic change in the reimplemented convertBody function or its caller.

This demo reproduces the problem:

Application.kt:

package com.example

import io.ktor.http.ContentType
import io.ktor.server.application.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import io.ktor.server.plugins.contentnegotiation.ContentNegotiation
import io.ktor.server.request.receive
import io.ktor.server.response.respondText
import io.ktor.server.routing.post
import io.ktor.server.routing.routing

fun main() {
    embeddedServer(Netty, port = 8080, host = "0.0.0.0", module = Application::module)
        .start(wait = true)
}

fun Application.module() {
    install(ContentNegotiation) {
        cvtA(ContentType.Application.Json)
        cvtB(ContentType.Application.Json)
    }
    routing {
        post("/") {
            val request = call.receive<CvtBResult>()

            call.respondText("ECHO: " + request.data)
        }
    }
}

CvtA.kt:

package com.example

import io.ktor.http.ContentType
import io.ktor.http.content.OutgoingContent
import io.ktor.http.content.TextContent
import io.ktor.http.withCharsetIfNeeded
import io.ktor.serialization.Configuration
import io.ktor.serialization.ContentConverter
import io.ktor.util.reflect.TypeInfo
import io.ktor.utils.io.ByteReadChannel
import io.ktor.utils.io.charsets.Charset
import io.ktor.utils.io.jvm.javaio.toInputStream
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext


data class CvtAResult(val data: String)

class CvtA : ContentConverter {
    override suspend fun deserialize(charset: Charset, typeInfo: TypeInfo, content: ByteReadChannel): Any? {
        return if (typeInfo.type == CvtAResult::class) {
            withContext(Dispatchers.IO) {
                val reader = content.toInputStream().reader(charset)
                CvtAResult(reader.readText())
            }
        } else {
            null
        }
    }

    override suspend fun serializeNullable(
        contentType: ContentType,
        charset: Charset,
        typeInfo: TypeInfo,
        value: Any?
    ): OutgoingContent? {
        return if (value != null && typeInfo.type == CvtAResult::class) {
            TextContent((value as CvtAResult).data, contentType.withCharsetIfNeeded(charset))
        } else {
            null
        }
    }
}

/**
 * Register a converter for org.JSON objects.
 */
fun Configuration.cvtA(
    contentType: ContentType = ContentType.Application.Json,
) {
    register(contentType, CvtA())
}

CvtB:kt:

package com.example

import io.ktor.http.ContentType
import io.ktor.http.content.OutgoingContent
import io.ktor.http.content.TextContent
import io.ktor.http.withCharsetIfNeeded
import io.ktor.serialization.Configuration
import io.ktor.serialization.ContentConverter
import io.ktor.util.reflect.TypeInfo
import io.ktor.utils.io.ByteReadChannel
import io.ktor.utils.io.charsets.Charset
import io.ktor.utils.io.jvm.javaio.toInputStream
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext


data class CvtBResult(val data: String)

class CvtB : ContentConverter {
    override suspend fun deserialize(charset: Charset, typeInfo: TypeInfo, content: ByteReadChannel): Any? {
        return if (typeInfo.type == CvtBResult::class) {
            withContext(Dispatchers.IO) {
                val reader = content.toInputStream().reader(charset)
                CvtBResult(reader.readText())
            }
        } else {
            null
        }
    }

    override suspend fun serializeNullable(
        contentType: ContentType,
        charset: Charset,
        typeInfo: TypeInfo,
        value: Any?
    ): OutgoingContent? {
        return if (value != null && typeInfo.type == CvtBResult::class) {
            TextContent((value as CvtBResult).data, contentType.withCharsetIfNeeded(charset))
        } else {
            null
        }
    }
}

/**
 * Register a converter for org.JSON objects.
 */
fun Configuration.cvtB(
    contentType: ContentType = ContentType.Application.Json,
) {
    register(contentType, CvtB())
}

Test request can be made via curl:

curl -i http://localhost:8080/  -H "Content-Type: application/json" --data '{}'

I expected the following output (behaviour of Ktor 2.1.3):

HTTP/1.1 200 OK
Content-Length: 8
Content-Type: text/plain; charset=UTF-8

ECHO: {}

However, with Ktor 2.2.1 this is printed:

HTTP/1.1 415 Unsupported Media Type
Content-Length: 0

As for the parallel use of org.json and kotlinx.serialization: I've previously used org.json only and then introduced kotlinx.serialization for handling data with static schema. I've now noticed that kotlinx.serialization.json provides JsonElement which appears to be able to handle data without a fixed schema. I might therefore switch away from using org.json entirely and thus this issue wouldn't affect me anymore.

{...} (tailcard) does not match URLs ending with '/'

CORS: `allowHost` without the second argument doesn't allow the secure host

RequestConnectionPoint should implement toString()

Other

CIO: Unable to perform WebSocket upgrade when Content-Type header is sent in the request

https://youtrack.jetbrains.com/issue/KTOR-6366

Hello, I have a web server that is built using Ktor, and is deployed on Heroku, In this server I configured WebSockets and created the following route:

fun Application.configureSockets() {

    val json by inject<Json>()

    install(WebSockets) {
        pingPeriod = Duration.ofSeconds(15)
        timeout = Duration.ofSeconds(15)
        maxFrameSize = Long.MAX_VALUE
        masking = false
        contentConverter = KotlinxWebsocketSerializationConverter(json)
    }
}

fun Route.getEnrolledStudentSocketRoute() {

    val services by inject<ClassServices>()

    webSocket(application.href(EndPoint.Socket.Class.EnrolledStudents())) {

        val classId = call.parameters["classId"] ?: return@webSocket close(CloseReason(CloseReason.Codes.CANNOT_ACCEPT, "class id is missing"))

        services.getEnrolledStudents(classId).collectLatest { students ->
            sendSerialized(EnrolledStudentsDto(classId, students.map(Student::asDto)))
        }
    }
}

From the client side, I have an Android application with Ktor Client, which is configured to use websockets, I have this request:

internal fun HttpClientConfig<*>.configureWebSockets(json: Json) {
    WebSockets {
        contentConverter = KotlinxWebsocketSerializationConverter(json)
    }
}

override suspend fun getEnrolledStudents(classId: String) = channelFlow {

  val url = client.href(TeacherStudentPackagesApi.WebSocket.EnrolledStudents(classId))

  client.wss(url) {
      incoming.consumeEach {
          this@channelFlow.send(receiveDeserialized())
      }
  }
}

However, when I try to establish the connection, the server always throw this error:

java.lang.IllegalStateException: Unable to perform upgrade as it is not requested by the client: request should have Upgrade and Connection headers filled properly
	at io.ktor.server.cio.CIOApplicationResponse.respondOutgoingContent(CIOApplicationResponse.kt:114)
	at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invokeSuspend(BaseApplicationResponse.kt:317)
	at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invoke(BaseApplicationResponse.kt)
	at io.ktor.server.engine.BaseApplicationResponse$Companion$setupSendPipeline$1.invoke(BaseApplicationResponse.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:88)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invokeSuspend(DefaultTransform.kt:29)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.websocket.RoutingKt.respondWebSocketRaw(Routing.kt:293)
	at io.ktor.server.websocket.RoutingKt.access$respondWebSocketRaw(Routing.kt:1)
	at io.ktor.server.websocket.RoutingKt$webSocketRaw$2$1$1$1.invokeSuspend(Routing.kt:105)
	at io.ktor.server.websocket.RoutingKt$webSocketRaw$2$1$1$1.invoke(Routing.kt)
	at io.ktor.server.websocket.RoutingKt$webSocketRaw$2$1$1$1.invoke(Routing.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invokeSuspend(Route.kt:116)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.routing.Routing.executeResult(Routing.kt:190)
	at io.ktor.server.routing.Routing.interceptor(Routing.kt:64)
	at io.ktor.server.routing.Routing$Plugin$install$1.invokeSuspend(Routing.kt:140)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invokeSuspend(BaseApplicationEngine.kt:124)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invokeSuspend(CommonHooks.kt:45)
  at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:78)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:264)
	at io.ktor.server.application.hooks.CallFailed$install$1.invokeSuspend(CommonHooks.kt:44)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:123)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
	at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
	at io.ktor.server.cio.CIOApplicationEngine$handleRequest$2.invokeSuspend(CIOApplicationEngine.kt:239)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

The request log from the client side:

REQUEST: wss://<BASE_URL>/ws/class/students?classId=default
METHOD: HttpMethod(value=GET)
COMMON HEADERS
-> Accept: application/json
-> Accept-Charset: UTF-8
-> Content-Type: application/json
CONTENT HEADERS
-> Connection: upgrade
-> Sec-WebSocket-Key: YTA4OWRkNDQ1YjU0YmVmYQ==
-> Sec-WebSocket-Version: 13
-> Upgrade: websocket
BODY Content-Type: null
BODY START

BODY END

That is weird, other clients such as postman can establish a successful connection, but the Android Ktor client does not. I am using CIO engine on both server and client, I have tried other JVM engines on my client, but all didn't work. From the above request log, found that `Upgrade` and `Connection` headers are properly filled, so I can't understand where is the problem exactly, It seems like a bug for me, I also tried some 2.3.x older versions, the error still arises.

2.3.7

released 7th December 2023

Docs

Broken link in "Logging"

Documentation Restructuring: Welcome page

2.3.6

released 7th November 2023

Client

Darwin: EOFException when sending multipart data using Ktor 2.3.4

AndroidClientEngine cannot handle content length that exceeds Int range

Darwin: Even a coroutine Job is canceled network load keeps high

Ktor JS client unconfigurable logging in node

"Server sent a subprotocol but none was requested" when using Node WebSockets

Client unable to make subsequent requests after the network disconnection and connection when ResponseObserver is installed

https://youtrack.jetbrains.com/issue/KTOR-6252

Description:

When using the Ktor HTTP client library to download files, an issue has been observed where the client fails to recover after an interrupted download caused by a loss of internet connection. This behavior is particularly evident when the provided code snippet is used to perform the download.

Steps to Reproduce:

Establish a working internet connection.
Execute the provided code snippet for making an HTTP request to download a file using the Ktor HTTP client.
During the file download process, intentionally disrupt the internet connection.
Allow the client to handle the connection loss and observe its behavior.

Expected Behavior: Upon detecting a loss of connectivity during the file download process, the Ktor HTTP client should throw an appropriate exception (e.g., a network-related exception) to indicate the connection failure. Subsequent attempts to make new HTTP requests using the client, after restoring the internet connection, should work as expected. The client should no longer be stuck in an unrecoverable state and should be capable of handling further requests without hindrance.

Actual Behavior: Currently, when the internet connection is disrupted during the file download process, the Ktor HTTP client appears to enter an unrecoverable state. Subsequent attempts to use the client to make additional HTTP requests (even after the internet connection is restored) fail to complete successfully. The client seems to be stuck in a state where it cannot recover from the interrupted download.

Code Snippet:

try {
    val httpResponse: HttpResponse = httpClient.get(url) {
        onDownload { bytesSentTotal, contentLength ->
            // Progress tracking logic can be added here if needed 
        }
    }
    val responseBody: ByteArray = httpResponse.body()
    file.writeBytes (responseBody)
} catch (e: Exception) {
    // Exception handling logic can be added here
}

Exception Stack Trace:

java.net.SocketTimeoutException: timeout at com.android.okhttp.okio.Okio$3.newTimeoutException(Okio.java:214) at com.android.okhttp.okio.AsyncTimeout.exit(AsyncTimeout.java:263) at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:217) at com.android.okhttp.okio.RealBufferedSource.read(RealBufferedSource.java:51) at com.android.okhttp.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:395) at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:372) at java.io.BufferedInputStream.fill(BufferedInputStream.java:248) at java.io.BufferedInputStream.read1(BufferedInputStream.java:288) at java.io.BufferedInputStream.read(BufferedInputStream.java:347) at io.ktor.utils.io.jvm.javaio.ReadingKt$toByteReadChannel$1.invokeSuspend(Reading.kt:55) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:188) at java.net.SocketInputStream.read(SocketInputStream.java:143) at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readFromSocket(ConscryptEngineSocket.java:945) at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:909) at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.readUntilDataAvailable(ConscryptEngineSocket.java:824) at com.android.org.conscrypt.ConscryptEngineSocket$SSLInputStream.read(ConscryptEngineSocket.java:797) at com.android.okhttp.okio.Okio$2.read(Okio.java:138) at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:213) at com.android.okhttp.okio.RealBufferedSource.read(RealBufferedSource.java:51) at com.android.okhttp.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:395) at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:372) at java.io.BufferedInputStream.fill(BufferedInputStream.java:248) at java.io.BufferedInputStream.read1(BufferedInputStream.java:288) at java.io.BufferedInputStream.read(BufferedInputStream.java:347) at io.ktor.utils.io.jvm.javaio.ReadingKt$toByteReadChannel$1.invokeSuspend(Reading.kt:55) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Note:

The preceding stack trace remains consistent for both the unsuccessful download request and the subsequent requests that follow it.

Environment:

Ktor version: [2.3.3]
HttpClientEngine: Android
Operating System: [Android 11]

WebSockets (CIO): Connection Failure Due to Lowercase 'upgrade' in 'Connection: upgrade' Header

WinHttp: ArrayIndexOutOfBoundsException when sending WS frame with empty body

https://youtrack.jetbrains.com/issue/KTOR-6394

It appears that sending a frame with an empty body (e.g. a Frame.Text("")) crashes an internal coroutine that fails incoming channel collections.

The root cause is the following:

Caused by: kotlin.ArrayIndexOutOfBoundsException
    at 0   ???                                 7ff7ae9c8d62       kfun:kotlin.Throwable#<init>(){} + 98 
    at 1   ???                                 7ff7ae9c2f7f       kfun:kotlin.Exception#<init>(){} + 79 
    at 2   ???                                 7ff7ae9c31ef       kfun:kotlin.RuntimeException#<init>(){} + 79 
    at 3   ???                                 7ff7ae9c337f       kfun:kotlin.IndexOutOfBoundsException#<init>(){} + 79 
    at 4   ???                                 7ff7ae9c3a4f       kfun:kotlin.ArrayIndexOutOfBoundsException#<init>(){} + 79 
    at 5   ???                                 7ff7ae9f1028       ThrowArrayIndexOutOfBoundsException + 120 
    at 6   ???                                 7ff7aed892a7       Kotlin_Arrays_getByteArrayAddressOfElement + 23 
    at 7   ???                                 7ff7ae9b8ed1       kfun:kotlinx.cinterop#addressOf__at__kotlinx.cinterop.Pinned<kotlin.ByteArray>(kotlin.Int){}kotlinx.cinterop.CPointer<kotlinx.cinterop.ByteVarOf<kotlin.Byte>> + 177 
    at 8   ???                                 7ff7aed26b9d       kfun:io.ktor.client.engine.winhttp.internal.WinHttpWebSocket.sendFrame#internal + 2877 
    at 9   ???                                 7ff7aed255f4       kfun:io.ktor.client.engine.winhttp.internal.WinHttpWebSocket.$sendNextFrameCOROUTINE$0.invokeSuspend#internal + 2580 
    at 10  ???                                 7ff7ae9ce785       kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 1029

It turns out the WinHttpWebSocket tries to use buffer.addressOf(0) even when the body (buffer) is empty:
https://github.com/ktorio/ktor/blob/b34415dafca82090e408761788a9b7ffb24788d7/ktor-client/ktor-client-winhttp/windows/src/io/ktor/client/engine/winhttp/internal/WinHttpWebSocket.kt#L201

I think the same issue is probably true for receiving frames with empty bodies:
https://github.com/ktorio/ktor/blob/b34415dafca82090e408761788a9b7ffb24788d7/ktor-client/ktor-client-winhttp/windows/src/io/ktor/client/engine/winhttp/internal/WinHttpWebSocket.kt#L109

Note that it might be hard to notice due to https://youtrack.jetbrains.com/issue/KT-62794 (we need to use printStackTrace() to see the cause).

Reproducer:

fun main() {
    runBlocking {
        val http = HttpClient(WinHttp) {
            install(WebSockets)
        }
        val session = http.webSocketSession("wss://ws.postman-echo.com/raw")
        try {
            session.send(Frame.Text(""))
            
            // This fails with ArrayIndexOutOfBoundsException, not because of the received echoed frame,
            // but because of the failed send causing the incoming channel to fail (see sendFrame in the stacktrace)
            session.incoming.receive()
        } finally {
            session.close()
        }
    }
}

Docs

The generator adds the '-jvm' suffix to all dependencies

Generator

The newly generated project is having problems with gradle

Gradle Plugin

Outdated Gradle jib plubin does not support application/vnd.oci.image.index.v1+json media type

Server

CIO: getEngineHeaderValues() returns duplicated values

https://youtrack.jetbrains.com/issue/KTOR-6352

Description:
When using embeddedCIOServer and geting all headers with call.response.headers.allValues() returned values are not as expected (examples below). When switching to another engine(e.g. Netty) returned values are as expected.

Describe the bug:

Function allValues() uses abstract function getEngineHeaderValues(name: String) from io.ktor.server.cio.CIOApplicationResponse and its implementation in cio-server is pretty confusing and in my testing wrong. I'm setting custom header for a route response like this:

call.response.headers.append("Test-Header", "Test-Value")

when getting header values the output is

{Test-Header=[Test-Value, Test-Value, Test-Value]}

When adding additional header with the same name:

call.response.headers.append("Test-Header", "Test-Value2")

the output is:

{Test-Header=[Test-Value, Test-Value, Test-Value, Test-Value2, Test-Value, Test-Value, Test-Value, Test-Value2]}

Output when using netty server:

{Test-Header=[Test-Value]}

{Test-Header=[Test-Value, Test-Value2, Test-Value, Test-Value2]}

*Second output is also correct since there are 2 headers with 2 values but with the same key

To reproduce:
Create simple CIO and Netty servers, add headers on the route and get response headers via allValues() function.

Or clone the project https://github.com/Theanko1412/ktor-issue-demo

Additional information:

After looking at git history this function was changed in commit from 2017, I mocked old behavior of this function with allValuesFixed() in my demo project and it is working as expected.
The response is having correct values, problem is only in retrieving it with allValues()

Proposed solution:
Revert function to its previous state.

If someone can provide some more information why was this changed and how was current implementation meant to be used I would appreciate it. Currently don't see any other way to correctly retrieve all headers from a response other than this.

Issue seems to be pretty old but couldn't find any old issues about this, if this is duplicate I would appreciate if you can point me to the solution and reasons why am I getting triple header value.

YAML properties with literal value null cannot be read since 2.3.1

https://youtrack.jetbrains.com/issue/KTOR-6357

Hello,

since version 2.3.1 (I suspect it came with the change https://youtrack.jetbrains.com/issue/KTOR-5797, but that's just a guess), it is not possible anymore to access a YAML configuration value that is assigned an explicit null value via the propertyOrNull method.

Example:

ktor:
    application:
        modules:
            - com.example.ApplicationKt.module
    deployment:
        port: 8080
    sample:
        prop1: null
#        pro2: null

package com.example

import io.ktor.server.application.*
import io.ktor.server.response.*
import io.ktor.server.routing.*

fun main(args: Array<String>) {
    io.ktor.server.netty.EngineMain.main(args)
}

fun Application.module() {
    routing {
        get("/") {
            val config = call.application.environment.config

            call.application.log.info("prop1: ${config.propertyOrNull("ktor.sample.prop1") == null}")
            call.application.log.info("prop2: ${config.propertyOrNull("ktor.sample.prop2") == null}")

            call.respondText("OK")
        }
    }
}

With up to version 2.3.0, it runs fine, but starting at 2.3.1, it fails as follows:

ApplicationConfigurationException: Expected primitive or list at path ktor.sample.prop1, but was class net.mamoe.yamlkt.YamlNull

I suspect that's the case because, in the YamlConfig#propertyOrNull implementation the line val value = yaml[parts.last()] ?: return null does not work as intended because the evaluation of the expression yaml[parts.last()] yields an instance of YamlNull instead of null itself. Therefore, the early exit with the evlis operator ?: return null does not fire. But again, that's just a suspicion of mine.

I am not sure if this is entirely intentional or if it indicates a bug. If it does, I would be glad if you could fix it. My temporary workaround is commenting out configuration variables whose values should be null instead of explicitly assigning null because then it behaves as previously.