Changelog 2.1 version

2.1.3

released 28th October 2022

Client

Cookies: Invalid encoding of cookies' values since 1.4.0

Websockets: timeout doesn't cause closing of incoming and outgoing channels

https://youtrack.jetbrains.com/issue/KTOR-2430

I'm developing network App using Ktor 1.2.5 and Kotlin 1.3.61 in Android 8

When netty websocket server's wifi is turned off by manually (or Ethernet cable unplugged manually), any event or exception is not happened in opposite websocket client even something is sent. Which mean websocket client does not know session is lost.

On the contrary, when websocket client's wifi is turned off, webserver can catch it because closedException: ClosedSendChannelException is occured when something is sent.

Does the any way for websocket client to know "session is lost" when server's network is disconnect?
(When webserver's Ethernet cable is unplugged, websocket client cannot be even closed normally - In other words, final block cannot be called.-

Here is code

-----------------------------------------------------------------
 SERVER
-----------------------------------------------------------------
embeddedServer(Netty, port = 8080) {
    install(DefaultHeaders)
    install(CallLogging)
    install(WebSockets) {
        pingPeriod = Duration.ofSeconds(10)
        timeout = Duration.ofSeconds(5)
    }
    ...

    routing {
        webSocket("/test") { 
            try {
                incoming.consumeEach { frame ->
                    if(frame is Frame.Text) {
                        Log.d("test", frame.readText())
                    }
                }
            } finally {
                val reason: CloseReason? = closeReason.await()
                Log.d("test", reason.toString())
            }
        }
    }
}


private suspend fun sendSomething(data:String) {
    try {
        clients.send(Frame.Binary(true, data))
    }
    catch (closedException: ClosedSendChannelException) {
        Log.d("test", "Server ClosedSendChannelException is happened")
    }
}

-----------------------------------------------------------------
 CLIENT
-----------------------------------------------------------------
CoroutineScope(Dispatchers.IO).launch {
    try {
        client.ws(host = ip, port = 8080, path = "/test") {
            pingIntervalMillis = 1000*10
            timeoutMillis = 1000*5

            ...

            try {
                incoming.consumeEach { frame ->
                    if(frame is Frame.Text) {
                        Log.d("test", frame.readText())
                    }
                }
            } catch (e: ClosedReceiveChannelException) {
                Log.d("test", "ClosedSendChannelException is happened")
            } catch (e: Throwable) {
                Log.d("test", "ClosedSendChannelException is happened")
            } finally {
// This block cannot be called even websocket client run explicitly disconnect function, when webserver's Ethernet cable is unplugged. 
                val reason: CloseReason? = closeReason.await()
                Log.d("test", reason.toString())
            }
        }
    } catch (e: Exception) {
        if (e is NoRouteToHostException || e is ConnectException) {
            Log.d("test", "Exception")
        } else if (e.javaClass.`package`?.name.toString().startsWith("java.net")) {
            Log.d("test", "java.net")
        } else {
            Log.d("test", "unknown")
        }
    }
}

private suspend fun sendSomething(data:String) {
    try {
        clients.send(Frame.Binary(true, data))
    }
    catch (closedException: ClosedSendChannelException) {
        Log.d("test", "Client ClosedSendChannelException is happened")
    }
}

CIO: A request through a proxy server results in 403 from Cloudflare

Core

RFC 3986 recommendation for encoding URI is NOT followed

JS: window.location.origin returns null when executed in iframe via srcdoc attribute

ByteReadChannel is unable to read files with long lines

https://youtrack.jetbrains.com/issue/KTOR-2588

We're using ktor and stores session data in redis. To avoid problems with the contents of the session we're base64'ing the data before storing it. This results in a one line file, 16kb big. This data is read back using ByteReadChannel, which apparently can not handle such long lines.

Tested on 1.4.21 and 1.4.32.

Test to trigger the exception:

    @Test
    fun decodeSessionData() {
        val fileContent = ByteBufferChannelTest::class.java.getResource("/16kb-on-one-line.txt").readText()
        val channel = ByteReadChannel(fileContent.toByteArray())

        // ktor session storage is using channel.readUTF8Line() to read session data
        runBlocking {
            channel.readUTF8Line()
        }
    }

Stacktrace on 1.4.21:

io.ktor.utils.io.charsets.TooLongLineException: Line is longer than limit
	at io.ktor.utils.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invokeSuspend(ByteBufferChannel.kt:2080)
	at io.ktor.utils.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invoke(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.lookAheadSuspend$suspendImpl(ByteBufferChannel.kt:1825)
	at io.ktor.utils.io.ByteBufferChannel.lookAheadSuspend(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineToUtf8Suspend(ByteBufferChannel.kt:2064)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineToAscii(ByteBufferChannel.kt:1999)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineTo$suspendImpl(ByteBufferChannel.kt:2099)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineTo(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8Line$suspendImpl(ByteBufferChannel.kt:2103)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8Line(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteReadChannelKt.readUTF8Line(ByteReadChannel.kt:223)

Stacktrace on 1.4.32:

io.ktor.utils.io.charsets.TooLongLineException: Line is longer than limit
	at io.ktor.utils.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invokeSuspend(ByteBufferChannel.kt:2094)
	at io.ktor.utils.io.ByteBufferChannel$readUTF8LineToUtf8Suspend$2.invoke(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.lookAheadSuspend$suspendImpl(ByteBufferChannel.kt:1827)
	at io.ktor.utils.io.ByteBufferChannel.lookAheadSuspend(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineToUtf8Suspend(ByteBufferChannel.kt:2076)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineToAscii(ByteBufferChannel.kt:2011)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineTo$suspendImpl(ByteBufferChannel.kt:2113)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8LineTo(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8Line$suspendImpl(ByteBufferChannel.kt:2117)
	at io.ktor.utils.io.ByteBufferChannel.readUTF8Line(ByteBufferChannel.kt)
	at io.ktor.utils.io.ByteReadChannelKt.readUTF8Line(ByteReadChannel.kt:223)

Docs

Fix the Kweet sample

Highlight use of the dispose function for Multipart Uploads in ktor docs.

Server

Websockets timeout doesn't cause a close of a connection

https://youtrack.jetbrains.com/issue/KTOR-3504

When setting a timeout on a ktor server websocket, the connection does not get closed once the timeout happens, if the client does not respond. It works as intended, if the timout happens, because the client is too slow to repond to a ping frame. It however does not work as intended, if the timeeout happens because the client lost their internet connection (in my case simulated by unplugging the LAN cable on the client machine).

What I expect to happen: When I set a ping and timeout on a websocket connection and the client loses the connection to the server, the timeout occures and I can no longer send Frames on the outgoing channel and any receive on the incoming channel results in a ClosedReceiveChannelException

What is actually happening: Once the client loses a connection, the timeout occurs, but a receive call is blocking as if the connection was still open. Sending frames on the outgoing channel does not result in an exception either.

What I think is the reason for this behaviour: Once the timeout occurs, the server sends a Close frame to the client and waits for the clients close acknowledgement. Since the client lost its connection, the close is not acknowledged and the server keeps the connection open.

This issue can be reproduced with the following code (connect to the websocket and remove the network cable between the server and the client):

fun main() {
    embeddedServer(Netty, port = 8080, host = "0.0.0.0") {
        install(WebSockets) {
            pingPeriod = Duration.ofSeconds(1)
            timeout = Duration.ofSeconds(1)
        }
        routing {
            webSocket("/socket") {
                print("Connection")
                while(true) {
                    withTimeoutOrNull(1000) {
                        println(incoming.receive())
                    }
                    outgoing.send(Frame.Text("test"))
                }
            }
        }
    }.start(wait = true)
}

I would expect that after the client lost the connection, either the incoming.receive() or the outgoing.send() should throw an exception, but neither happens.

HOCON: CLI parameters don't override custom properties since 2.1.0

Autoreloading: "Flow invariant is violated" error since Ktor 2.0.3

https://youtrack.jetbrains.com/issue/KTOR-4926

When trying to upgrade from Ktor 2.0.2 to 2.0.3 (and all the way to 2.1.2), we started encountering strange Flow invariant violations. The stacktrace looks something like this:

java.lang.IllegalStateException: Flow invariant is violated:
		Flow was collected in [io.sentry.kotlin.SentryContext@51f480a5, com.corp.app.ApplicationKt$module$2$1$1$invokeSuspend$$inlined$CoroutineExceptionHandler$1@4652be27, CoroutineId(6), \"coroutine#6\":ScopeCoroutine{Active}@4f97a3bd, io.ktor.server.engine.ClassLoaderAwareContinuationInterceptor@3b8c8a07],
		but emission happened in [io.sentry.kotlin.SentryContext@51f480a5, com.corp.app.ApplicationKt$module$2$1$1$invokeSuspend$$inlined$CoroutineExceptionHandler$1@4652be27, CoroutineId(6), \"coroutine#6\":ScopeCoroutine{Active}@4f97a3bd, io.ktor.server.engine.ClassLoaderAwareContinuationInterceptor@3b8c8a07].
		Please refer to 'flow' documentation or use 'flowOn' instead
	at kotlinx.coroutines.flow.internal.SafeCollector_commonKt.checkContext(SafeCollector.common.kt:85)
	at kotlinx.coroutines.flow.internal.SafeCollector.checkContext(SafeCollector.kt:106)
	at kotlinx.coroutines.flow.internal.SafeCollector.emit(SafeCollector.kt:83)
	at kotlinx.coroutines.flow.internal.SafeCollector.emit(SafeCollector.kt:66)
	at com.corp.app.extensions.aws.SqsKt$getFlowFor$1.invokeSuspend(Sqs.kt:33)
	at \u0008\u0008\u0008(Coroutine boundary.\u0008(\u0008)
	at com.corp.app.ApplicationKt$module$2$1$1$1.invokeSuspend(Application.kt:172)
	at \u0008\u0008\u0008(Coroutine creation stacktrace.\u0008(\u0008)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:47)
	at kotlinx.coroutines.BuildersKt.launch$default(Unknown Source)
	at com.corp.app.ApplicationKt$module$2$1$1.invokeSuspend(Application.kt:171)
	at com.corp.app.ApplicationKt$module$2$1$1.invoke(Application.kt)
	at com.corp.app.ApplicationKt$module$2$1$1.invoke(Application.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
	at kotlinx.coroutines.SupervisorKt.supervisorScope(Supervisor.kt:61)
	at com.corp.app.ApplicationKt$module$2.invokeSuspend(Application.kt:166)
[...]

After trying all kinda ways trying to fix coroutine contexts, I found KTOR-4164 in the changelog and tried setting ktor.development = false in my config which significantly changes behavior on this (makes it much less reproducible on my side). I think this may be an unintended regression of the "fix" for the other issue?

Just in case it's an issue with my code, the method in question looks like this (uses the AWS Java SDK):

fun <T> SqsAsyncClient.getFlowFor(queueUrl: String, klass: Class<T>) = flow {
    val queue = this@getFlowFor
    val flow = this

    while (currentCoroutineContext().isActive) {
        val response = queue.receiveMessage(
            ReceiveMessageRequest.builder()
                .queueUrl(queueUrl)
                .maxNumberOfMessages(10)
                .waitTimeSeconds(20)
                .build()
        ).await()
        if (!response.hasMessages()) continue

        val messages = response.messages()
        messages
            .map { json.readValue(it.body(), klass) }
            .forEach { flow.emit(it) }
    }
}

SensitivityWatchEventModifier - Move the reflection call of this modifier out from the Ktor Core

DefaultHeaders: a header is duplicated in a StatusPages's handler

Netty HTTP/2: response headers contain ":status" header and that leads to IllegalHeaderNameException in the ConditionalHeaders plugin

Maven: ktor-server-test-host-jvm causes dependency error starting from Ktor 2.0.3

Autoreloading: ClassCastException when retrieving plugins in testApplication

https://youtrack.jetbrains.com/issue/KTOR-4729

Hello, while working on creating a Ktor plugin and setting up tests for it, I've encountered the following bug:

When creating a test that directly retrieves a plugin (either via the Application.plugin() function or when using custom extension functions over Application), the code crashes with a ClassCastException. While the class is the exact same, the instance of the class returned by the test Ktor application comes from another classloader, leading to the exceptions you can see below. It seems to be related to the development mode's class reloading feature interfering somewhere along the line.

Repro

Here's a GitHub repository to repro this issue: https://github.com/utybo/cce_ktor_test
Here's a GitHub Actions run with the failing tests: https://github.com/utybo/cce_ktor_test/runs/7724799083?check_suite_focus=true#step:5:34

Here's an example, let's take this plugin:

class MyCalculatorPlugin {
    class Configuration
    companion object Plugin : BaseApplicationPlugin<ApplicationCallPipeline, Configuration, MyCalculatorPlugin> {
        override val key = AttributeKey<MyCalculatorPlugin>("MyCalculatorPlugin")

        override fun install(
            pipeline: ApplicationCallPipeline,
            configure: Configuration.() -> Unit
        ): MyCalculatorPlugin {
            return MyCalculatorPlugin()
        }

    }

    fun add(x: Int, y: Int): Int {
        return x + y
    }
}

val Application.myCalculator get() = plugin(MyCalculatorPlugin)

The following tests will fail:

@Test
fun `Test simple sum via 'myCalculator' utility property`() = testApplication {
    install(MyCalculatorPlugin)
    application {
        val result = myCalculator.add(1, 2) // <-- ClassCastException
        assertEquals(3, result)
    }
}

@Test
fun `Test simple sum via 'plugin' function`() = testApplication {
    install(MyCalculatorPlugin)
    application {
        val result = plugin(MyCalculatorPlugin).add(1, 2) // <-- ClassCastException
        assertEquals(3, result)
    }
}

Stack trace for the first test:

class org.example.ccektortest.MyCalculatorPlugin cannot be cast to class org.example.ccektortest.MyCalculatorPlugin (org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader 'app'; org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader io.ktor.server.engine.OverridingClassLoader$ChildURLClassLoader @68ad99fe)
java.lang.ClassCastException: class org.example.ccektortest.MyCalculatorPlugin cannot be cast to class org.example.ccektortest.MyCalculatorPlugin (org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader 'app'; org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader io.ktor.server.engine.OverridingClassLoader$ChildURLClassLoader @68ad99fe)
	at org.example.ccektortest.LibraryKt.getMyCalculator(Library.kt:28)
	at org.example.ccektortest.LibraryTest$Test simple sum via 'myCalculator' utility property$1$1.invoke(LibraryTest.kt:17)
	at org.example.ccektortest.LibraryTest$Test simple sum via 'myCalculator' utility property$1$1.invoke(LibraryTest.kt:16)
	at io.ktor.server.engine.internal.CallableUtilsKt.executeModuleFunction(CallableUtils.kt:51)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:334)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:333)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:358)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.launchModuleByName(ApplicationEngineEnvironmentReloading.kt:333)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$launchModuleByName(ApplicationEngineEnvironmentReloading.kt:32)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:321)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:312)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:340)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:312)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:149)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:279)
	at io.ktor.server.testing.TestApplicationEngine.start(TestApplicationEngine.kt:125)
	at io.ktor.server.engine.ApplicationEngine$DefaultImpls.start$default(ApplicationEngine.kt:68)
	at io.ktor.server.testing.TestApplicationKt.testApplication(TestApplication.kt:267)
	at org.example.ccektortest.LibraryTest.Test simple sum via 'myCalculator' utility property(LibraryTest.kt:14)
	[snip]

Stack trace for the second test:

class org.example.ccektortest.MyCalculatorPlugin cannot be cast to class org.example.ccektortest.MyCalculatorPlugin (org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader 'app'; org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader io.ktor.server.engine.OverridingClassLoader$ChildURLClassLoader @5b057c8c)
java.lang.ClassCastException: class org.example.ccektortest.MyCalculatorPlugin cannot be cast to class org.example.ccektortest.MyCalculatorPlugin (org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader 'app'; org.example.ccektortest.MyCalculatorPlugin is in unnamed module of loader io.ktor.server.engine.OverridingClassLoader$ChildURLClassLoader @5b057c8c)
	at org.example.ccektortest.LibraryTest$Test simple sum via 'plugin' function$1$1.invoke(LibraryTest.kt:26)
	at org.example.ccektortest.LibraryTest$Test simple sum via 'plugin' function$1$1.invoke(LibraryTest.kt:25)
	at io.ktor.server.engine.internal.CallableUtilsKt.executeModuleFunction(CallableUtils.kt:51)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:334)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$launchModuleByName$1.invoke(ApplicationEngineEnvironmentReloading.kt:333)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:358)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.launchModuleByName(ApplicationEngineEnvironmentReloading.kt:333)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$launchModuleByName(ApplicationEngineEnvironmentReloading.kt:32)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:321)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:312)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:340)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:312)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:149)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:279)
	at io.ktor.server.testing.TestApplicationEngine.start(TestApplicationEngine.kt:125)
	at io.ktor.server.engine.ApplicationEngine$DefaultImpls.start$default(ApplicationEngine.kt:68)
	at io.ktor.server.testing.TestApplicationKt.testApplication(TestApplication.kt:267)
	at org.example.ccektortest.LibraryTest.Test simple sum via 'plugin' function(LibraryTest.kt:23)
	[snip]

Workaround

Disable development mode in tests by adding the following line inside the testApplication block:

environment { developmentMode = false }