Changelog 1.4 version

1.4.3

released 1st December 2020

Client

iOS client fails with CoroutinesInternalError when Logging is used

https://youtrack.jetbrains.com/issue/KTOR-924

Creating a simple HttpClient with the Logging feature included fails with the attached stack trace. The failure occurs both for real web calls in-app, as well as for a unit test using MockEngine. Both the test and the in-app call succeed if the logging configuration is removed.

I have a simple reproducer project attached, and also available at https://github.com/russhwolf/Ktor14

Note that my test configuration is using version 1.3.9-native-mt of kotlinx.coroutines, because version 1.3.9 leads to KTOR-915 / KTOR-919

kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[EventLoopImpl@36817bc8, Continuation @ $split$lambda-0$<anonymous>_3COROUTINE$2]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
	at kotlin.Error#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:14)
	at kotlinx.coroutines.CoroutinesInternalError#<init>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Exceptions.common.kt:28)
	at kotlinx.coroutines.DispatchedTask#handleFatalException(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:93)
	at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:64)
	at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
	at kotlinx.coroutines#runEventLoop(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:80)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67)
	at kotlinx.coroutines#runBlocking(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50)
	at com.example#runBlocking(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:8)
	at com.example#runBlocking$default(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:7)
	at com.example.ApiClientTest#mockWebCall(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:20)
	at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.invoke#internal(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
	at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.$<bridge-UNNN>invoke(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
	at kotlin.native.internal.test.BaseClassSuite.TestCase#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestSuite.kt:85)
	at kotlin.native.internal.test.TestRunner.run#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:236)
	at kotlin.native.internal.test.TestRunner.runIteration#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:258)
	at kotlin.native.internal.test.TestRunner#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:273)
	at kotlin.native.internal.test#testLauncherEntryPoint(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:22)
	at kotlin.native.internal.test#main(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:26)
	at <global>.Konan_start(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:25)
	at <global>.Init_and_run_start(Unknown Source)
	at <global>.start(Unknown Source)
	at kotlin.Exception#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:25)
	at kotlin.RuntimeException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:36)
	at kotlinx.coroutines.CompletionHandlerException#<init>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Exceptions.common.kt:13)
	at kotlinx.coroutines.JobSupport.notifyCompletion#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:365)
	at kotlinx.coroutines.JobSupport.completeStateFinalization#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:326)
	at kotlinx.coroutines.JobSupport.finalizeFinishingState#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:241)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:922)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:875)
	at kotlinx.coroutines.JobSupport#makeCompletingOnce(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:840)
	at kotlinx.coroutines.AbstractCoroutine#resumeWith(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/AbstractCoroutine.kt:111)
	at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/coroutines/ContinuationImpl.kt:43)
	at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:53)
	at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
	at kotlin.Throwable#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23)
	at kotlin.Exception#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23)
	at kotlin.RuntimeException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34)
	at kotlin.native.concurrent.InvalidMutabilityException#<init>(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:22)
	at <global>.ThrowInvalidMutabilityException(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:92)
	at <global>.MutationCheck(Unknown Source)
	at kotlinx.coroutines.AwaitAll.AwaitAllNode.<set-handle>#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:94)
	at kotlinx.coroutines.AwaitAll.await#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:68)
	at kotlinx.coroutines.$awaitAllCOROUTINE$1#invokeSuspend(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:38)
	at kotlinx.coroutines#awaitAll@kotlin.collections.Collection<kotlinx.coroutines.Deferred<0:0>>(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/Await.kt:37)
	at io.ktor.util.$split$lambda-0COROUTINE$3.invokeSuspend#internal(/opt/buildAgent/work/a85294440dc5c6e/ktor-utils/common/src/io/ktor/util/ByteChannels.kt:29)
	at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/coroutines/ContinuationImpl.kt:30)
	at kotlinx.coroutines.DispatchedTask#run(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/internal/DispatchedTask.kt:39)
	at kotlinx.coroutines.EventLoopImplBase#processNextEvent(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:274)
	at kotlinx.coroutines#runEventLoop(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:80)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67)
	at kotlinx.coroutines#runBlocking(/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50)
	at com.example#runBlocking(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:8)
	at com.example#runBlocking$default(/Users/russell/IdeaProjects/Ktor14/shared/src/iosTest/kotlin/runBlocking.kt:7)
	at com.example.ApiClientTest#mockWebCall(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:20)
	at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.invoke#internal(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
	at com.example.$ApiClientTest$test$0.$mockWebCall$FUNCTION_REFERENCE$0.$<bridge-UNNN>invoke(/Users/russell/IdeaProjects/Ktor14/shared/src/commonTest/kotlin/ApiClientTest.kt:19)
	at kotlin.native.internal.test.BaseClassSuite.TestCase#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestSuite.kt:85)
	at kotlin.native.internal.test.TestRunner.run#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:236)
	at kotlin.native.internal.test.TestRunner.runIteration#internal(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:258)
	at kotlin.native.internal.test.TestRunner#run(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/TestRunner.kt:273)
	at kotlin.native.internal.test#testLauncherEntryPoint(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:22)
	at kotlin.native.internal.test#main(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:26)
	at <global>.Konan_start(/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/internal/test/Launcher.kt:25)
	at <global>.Init_and_run_start(Unknown Source)

CIO: client engine exceptions are both logged and thrown

https://youtrack.jetbrains.com/issue/KTOR-1127

In the CIO client engine exceptions like SocketTimeoutException are logged to standard error by the coroutines' uncaught exception handler and thrown by client.get().

import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*

suspend fun main(): Unit = withContext(Dispatchers.Default) {
    HttpClient(CIO) {
        install(HttpTimeout) {
            socketTimeoutMillis = 5
        }
    }.use { client ->
        try {
            client.get<HttpResponse>("http://www.google.com").readText()
        } catch (e: Throwable) {
            println("Caught: $e")
        }
    }
}

Output:

Exception in thread "kotlinx.coroutines.DefaultExecutor" java.net.SocketTimeoutException
	at io.ktor.network.util.UtilsKt$withSocketTimeout$2.invokeSuspend(Utils.kt:23)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32)
	at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:188)
	at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:122)
	at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:111)
	at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:308)
	at kotlinx.coroutines.CancellableContinuationImpl.cancel(CancellableContinuationImpl.kt:177)
	at kotlinx.coroutines.CancellableContinuationImpl.parentCancelled$kotlinx_coroutines_core(CancellableContinuationImpl.kt:184)
	at kotlinx.coroutines.ChildContinuation.invoke(JobSupport.kt:1484)
	at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1511)
	at kotlinx.coroutines.JobSupport.tryMakeCancelling(JobSupport.kt:792)
	at kotlinx.coroutines.JobSupport.makeCancelling(JobSupport.kt:752)
	at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:668)
	at kotlinx.coroutines.JobSupport.cancelCoroutine(JobSupport.kt:655)
	at kotlinx.coroutines.TimeoutCoroutine.run(Timeout.kt:128)
	at kotlinx.coroutines.EventLoopImplBase$DelayedRunnableTask.run(EventLoop.common.kt:497)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.DefaultExecutor.run(DefaultExecutor.kt:68)
	at java.base/java.lang.Thread.run(Thread.java:832)

Caught: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=http://www.google.com/, socket_timeout=5] ms

Timeout feature: android engine throws Java's SocketTimeoutException instead of ConnectTimeoutException

HttpRedirect feature alters Location header value

CIO Engine's HttpClient may fail when trying to send large size binary data.

https://youtrack.jetbrains.com/issue/KTOR-1247

Ktor Version and Engine Used (client or server and name)

Client CIO, 1.4.1

Describe the bug

HttpClient with CIO Engine sometimes sends an invalid multipart/form-data request when it includes large size binary data. When it comes to invalid requests, it does not seem to contain a trailing multipart/form-data request boundary. (There doesn't seem to be enough content in the request body.)

java.nio.BufferOverflowException seems to occur in the following process. https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L123

This behavior is not reproduced in the case of HttpClient with Apache Engine.

To Reproduce

Sample code is below.

@KtorExperimentalAPI
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    routing {
        post("/multipart") {
            val multipartData = call.receiveMultipart()
            multipartData.forEachPart {
                if (it is PartData.FileItem) {
                    println(
                        withContext(Dispatchers.IO) {
                            it.streamProvider().readAllBytes().size
                        }
                    )
                }
            }
            call.respondText("test")
        }
        get("/execute/cio") {
            repeat(10) {
                request(HttpClient(CIO))
            }
            call.respondText("success")
        }
        get("/execute/apache") {
            repeat(10) {
                request(HttpClient(Apache))
            }
            call.respondText("success")
        }
    }
}

suspend fun request(client: HttpClient) {
    val body = formData {
        val headers = headersOf(
            HttpHeaders.ContentDisposition,
            ContentDisposition.File
                .withParameter(ContentDisposition.Parameters.Name, "file")
                .withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
                .toString()
        )
        appendInput("file", headers) {
            // dummy 10MB binary file
            ByteArray(10_000_000).inputStream().asInput()
        }
    }
    client.use {
        it.submitFormWithBinaryData<Unit>(
            url = "http://localhost:8080/multipart",
            formData = body
        )
    }
}

(full code is in https://github.com/ma2Cta/ktor-multipart-request-test/blob/main/src/Application.kt )

Start the server.
Run $ curl http://localhost:8080/execute/apache a few times.

10 times 10000000 will be printed as expected.

Run $ curl http://localhost:8080/execute/cio a few times.

You won't get 10000000 10 times, and you'll probably get a number less than 10000000 the first few times

Expected

The correct request is sent even in the case of CIO Engine.

Client: NPE in FormDataContentKt -> Input.copyTo

https://youtrack.jetbrains.com/issue/KTOR-1349

I've received the following exception report for an Android app. It was sporadic and I cannot reproduce the problem. The problem seems to lie within Ktor itself so I'll simply share what I can.

FATAL EXCEPTION: k.a0 Dispatcher = okhttp3.OkHttpClient Dispatcher
java.lang.NullPointerException
	at h.b.a.g.j.b$a.l(Unknown Source:58) = io.ktor.client.request.forms.FormDataContentKt$copyTo$2.invokeSuspend(java.lang.Object)
	at kotlin.c0.k.a.a.n(Unknown Source:9) = kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(java.lang.Object)
	at kotlinx.coroutines.a1.run(Unknown Source:93)
	at kotlinx.coroutines.d3.b.b0(Unknown Source:0)
	at kotlinx.coroutines.d3.b$a.c(Unknown Source:14)
	at kotlinx.coroutines.d3.b$a.m(Unknown Source:28)
	at kotlinx.coroutines.d3.b$a.run(Unknown Source:0)

This is the relevant mapping:

io.ktor.client.request.forms.FormDataContentKt$copyTo$2 -> h.b.a.g.j.b$a:
    io.ktor.utils.io.WriterSuspendSession p$ -> p
    java.lang.Object L$0 -> q
    io.ktor.utils.io.core.Input $this_copyTo -> s
    int label -> r
    java.lang.Object invoke(java.lang.Object,java.lang.Object) -> M
    kotlin.coroutines.Continuation create(java.lang.Object,kotlin.coroutines.Continuation) -> d
    java.lang.Object invokeSuspend(java.lang.Object) -> l

The relevant code from Ktor: https://github.com/ktorio/ktor/blob/a6d68420ee0b272cd09d235057427a6601736487/ktor-client/ktor-client-core/common/src/io/ktor/client/request/forms/FormDataContent.kt#L154

private suspend fun Input.copyTo(channel: ByteWriteChannel) {
    if (this is ByteReadPacket) {
        channel.writePacket(this)
        return
    }

    channel.writeSuspendSession {
        while (!this@copyTo.endOfInput) {
            tryAwait(1)
            val buffer = request(1)!!                                   // <-- likely cause
            val size = this@copyTo.readAvailable(buffer)

            if (size < 0) continue
            written(size)
        }
    }
}

Core

Experimental API and compatibility guarantees

https://youtrack.jetbrains.com/issue/KTOR-1035

Problem

Currently, we have a public commitment to not break any API and ABI. This is generally good. The problem is that it makes it almost impossible for a new experimental feature/API to grow and entirely stops evolution since major versions are released too slowly.

Proposed solution

The idea is to relax guarantees for all API that is marked with experimental annotations similar to Kotlin.

Rules

in a patch version: no changes allowed, deprecations with level WARNING could be introduced for existing API.
in a minor version: both source and binary changes allowed except for complete removal. All changes should include migration/deprecation. Hidden deprecated APIs should be introduced for binary compatibility.
in a major version: any changes including complete removal but only through the deprecation cycle.

Requirements for experimental API:

experimental API should be marked with an experimental annotation requiring opt-in
an experimental annotation should have level ERROR and should have a reasonable and clear description
for a single or group of new experimental functions/classes/APIs having the same purpose and/or related functionality, it is required to introduce a new experimental annotation

Note: the consequence is that you can't use KtorExperimentalAPI annotation for new API. So one should always create new experimental annotations for now. The other consequence is that we should revisit all KtorExperimentalAPI usages and either stabilize them or migrate to new experimental annotations.

Requirements for experimental API annotation:

An experimental annotation class should:

require opt-in with level ERROR (@RequiresOptIn(level = ERROR));
have a reasonable and clear description;
have a reference to a ktor ticket that introduced this API and/or describes why the API is experimental and that is useful for tracking the concrete experimental feature status;
have retention BINARY, unless there is a reason to have RUNTIME;
be documented;
contain a disclamer in the kdoc about the risk of using experimental features.

Note: RUNTIME retention is not recommended since the ability to analyze it in runtime makes it more difficult to remove the annotation due to indirect and unobvious references. Retention SOURCE is not acceptable because it is how @RequiresOptIn works in Kotlin.

Experimental API Deprecation cycle

A deprecation cycle for an API that is intended to be removed or changed consist of the following steps:

In any release, make the API deprecated with level WARNING by applying one of the following:

@Deprecated annotation with level WARNING, provide a clear message describing what will be changed and what exact migration is required, provide a "replace with"
a new annotation requiring opt-in with level WARNING and the description

In the next minor release, promote the deprecation level from WARNING to ERROR
In the next minor release after step 2, change the deprecation level from ERROR to HIDDEN, keep the implementation working despite that it is hidden (to keep binary compatibility)
In the next major release, remove the old API

Please note that using an opt-in annotation for deprecation doesn't provide the ability to write a "replace with". Thus, it only makes sense to use it for removing old API. A good example is kotlinx.coroutines ObsoleteCoroutinesApi

If some of the rules are impossible to comply (for example, it is impossible to provide a "replace with" action or impossible to keep a hidden function working), an author of change should initiate a discussion at least with PR reviewer. When decided, it could be also discussed in the team or even publically for a quite popular experimental API. If it was decided to skip some of the requirements, it is important to document that decision and reasons in the PR.

KtorExperimentalAPI deprecation

@KtorExperimentalAPI annotation is going to be deprecated in the following stages:

Revisit all existing API marked with the annotation and stabilize when possible without breaking changes in a minor version(s). Mark the annotation deprecated with level WARNING.
In major version (2.0.0), revisit again. When impossible to stabilize, introduce new experimental annotations and use them instead. Ensure no KtorExperimentalAPI usages. Promote its deprecation to level ERROR.
In the next major version (3.0.0) deprecate it with level HIDDEN. Use text search to ensure no text occurrences.
In the next major version (4.0.0) remove it.

The reason why we deprecate it is that it introduces a way of "anonymous" experiment. An experimental API should be introduced with a clear reason and plan. A plan should cover only one particular feature/functionality. An experimental feature should have a more or less clear lifecycle.

The other disadvantage of this annotation is that there is no easy way to de-opt-in users from it. Also, it invites most brave users to opt-in globally for the whole project (just as we did for ExperimentalCoroutinesApi). This is very bad because in this case experimental usages will become uncontrolled and the user doesn't realize anymore what danger it is. So to prevent such problems, we should stop using it and migrate to a better approach.

Example

Here is an example of an experimental API:

/**
 * API marked with this annotation is a part of experimental Z feature. 
 * The feature preview provides the ability to work with Z. 
 * Please see KTOR-number for details and stabilization plan.
 * 
 * This is an API preview and it's not recommended for production development 
 * since it could be changed or removed in the future and 
 * not guaranteed to remain functional and its behaviour may also change in any non-patch release.
 */
@RequiresOptIn("Z feature is experimental and not recommended for production development.", level = ERROR)
@Retention(BINARY)
annotation class ZFeature

/**
 * Z feature's context
 */
@ZFeature
class ZContext

/** 
 * Configures Z feature and invokes [block] with running instance.
 */
@ZFeature
fun withZ(block: () -> Unit) = ...

Input.readTextExactBytes(n) on empty input different behavior per platform

ktor-io: JVM shared function decrease performance starting from 1.4.0

Infrastructure

Add build parameter to build ktor with JVM IR compiler

CI does not trigger build on PR to samples repo

Invalid publication 'kotlinMultiplatform' for task :publishKotlinMultiplatformPublicationToMavenLocal on master

Server

HTTP/2 push fails with netty engine

https://youtrack.jetbrains.com/issue/KTOR-800

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/350

The problem reproduces with http2-push from https://github.com/ktorio/ktor-samples

Run ./gradlew :http2-push:run
Navigate to https://localhost:8443/

Resulting exception:

2018-03-07 14:47:07.089 [nettyWorkerPool-3-2] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.IllegalArgumentException: Stream no longer exists: 3
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.requireStream(DefaultHttp2ConnectionEncoder.java:343)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writePushPromise(DefaultHttp2ConnectionEncoder.java:279)
        at io.ktor.server.netty.http2.NettyHttp2Handler.startHttp2PushPromise$ktor_server_netty(NettyHttp2Handler.kt:104)
        at io.ktor.server.netty.http2.NettyHttp2ApplicationResponse$push$1.run(NettyHttp2ApplicationResponse.kt:49)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
2018-03-07 14:47:07.093 [nettyWorkerPool-3-2] WARN  i.n.util.concurrent.DefaultPromise - An exception was thrown by io.ktor.server.netty.http2.NettyHttp2Handler$startHttp2PushPromise$1.operationComplete()
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Stream no longer exists: 3
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at io.ktor.server.netty.http2.NettyHttp2Handler$startHttp2PushPromise$1.operationComplete(NettyHttp2Handler.kt:109)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
        at io.ktor.server.netty.http2.NettyHttp2Handler.startHttp2PushPromise$ktor_server_netty(NettyHttp2Handler.kt:108)
        at io.ktor.server.netty.http2.NettyHttp2ApplicationResponse$push$1.run(NettyHttp2ApplicationResponse.kt:49)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Stream no longer exists: 3
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.requireStream(DefaultHttp2ConnectionEncoder.java:343)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writePushPromise(DefaultHttp2ConnectionEncoder.java:279)
        at io.ktor.server.netty.http2.NettyHttp2Handler.startHttp2PushPromise$ktor_server_netty(NettyHttp2Handler.kt:104)
        ... 7 common frames omitted

Request headers exceeding expected threshold are not handled correctly

Cannot receive content via jackson negotiator since 1.4.2

https://youtrack.jetbrains.com/issue/KTOR-1286

It seems like this commit broke the Jackson content negotiator, now when calling call.receive() the following exception occurs:

java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
	at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302)
	at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1)
	at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30)
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20)
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20)
	at io.ktor.jackson.JacksonConverter.convertForReceive(JacksonConverter.kt:55)
	at io.ktor.features.ContentNegotiation$Feature$install$3.invokeSuspend(ContentNegotiation.kt:175)
	at io.ktor.features.ContentNegotiation$Feature$install$3.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.request.ApplicationReceiveFunctionsKt.receive(ApplicationReceiveFunctions.kt:110)
	at net.octyl.ourtwobe.api.ApiServerKt$module$7$2$1$1.invokeSuspend(ApiServer.kt:232)
	at net.octyl.ourtwobe.api.ApiServerKt$module$7$2$1$1.invoke(ApiServer.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.routing.Routing.executeResult(Routing.kt:148)
	at io.ktor.routing.Routing.interceptor(Routing.kt:35)
	at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:101)
	at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194)
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:100)
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:140)
	at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139)
	at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:55)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:38)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)

Sessions + SSL (Netty)

https://youtrack.jetbrains.com/issue/KTOR-1292

edit: https://github.com/dm6801/ktor_sessions_ssl

when trying to use Session feature with cookies

    install(Sessions) {
        cookie<MySession>("MY_SESSION")
    }

accessing https:/localhost:443/ throws:

java.lang.UnsupportedOperationException: null
	(Coroutine boundary)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:55)
	(Coroutine creation stacktrace)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:188)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:38)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:821)
Caused by: java.lang.UnsupportedOperationException: null
	at io.ktor.server.netty.http2.NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64)
	at io.ktor.sessions.SessionTransportCookie.receive(SessionTransportCookie.kt:28)
	at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:204)
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:59)
	at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)

relevant line NettyHttp2ApplicationRequest.getCookies(NettyHttp2ApplicationRequest.kt:64):

override val cookies: RequestCookies
    get() = throw UnsupportedOperationException()

http://localhost:8080/ - works

Netty HTTP/2 HEAD response hangs

Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.

"Wrong HEX escape": gracefully handle invalid URLs

https://youtrack.jetbrains.com/issue/KTOR-1308

We recently got some alerts in our webserver because ktor threw an uncaught exception. Ideally these are handled and a 400 is sent back.

Here is the exception: Wrong HEX escape: %uf, in /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/winnt/win.ini

Stacktrace:

at io.ktor.http.CodecsKt.decodeImpl (Codecs.kt:197)
at io.ktor.http.CodecsKt.decodeScan (Codecs.kt:146)
at io.ktor.http.CodecsKt.decodeURLPart (Codecs.kt:140)
at io.ktor.http.CodecsKt.decodeURLPart$default (Codecs.kt:139)
at io.ktor.routing.RoutingResolveContext.parse (RoutingResolve.kt:70)
at io.ktor.routing.RoutingResolveContext.<init> (RoutingResolve.kt:49)
at io.ktor.routing.Routing.interceptor (Routing.kt:31)
at io.ktor.routing.Routing$Feature$install$1.invokeSuspend (Routing.kt:99)
at io.ktor.routing.Routing$Feature$install$1.invoke (Routing.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend (ContentNegotiation.kt:107)
at io.ktor.features.ContentNegotiation$Feature$install$1.invoke (ContentNegotiation.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend (DefaultEnginePipeline.kt:120)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke (DefaultEnginePipeline.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop (PipelineContext.kt:318)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed (PipelineContext.kt:163)
at io.ktor.util.pipeline.SuspendFunctionGun.execute (PipelineContext.kt:183)
at io.ktor.util.pipeline.Pipeline.execute (Pipeline.kt:27)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend (NettyApplicationCallHandler.kt:40)
at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke (NettyApplicationCallHandler.kt)
at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched (Undispatched.kt:55)
at kotlinx.coroutines.CoroutineStart.invoke (CoroutineStart.kt:111)
at kotlinx.coroutines.AbstractCoroutine.start (AbstractCoroutine.kt:158)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch (Builders.common.kt:54)
at kotlinx.coroutines.BuildersKt.launch (Unknown Source)
at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest (NettyApplicationCallHandler.kt:30)
at io.ktor.server.netty.NettyApplicationCallHandler.channelRead (NettyApplicationCallHandler.kt:24)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead (AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.access$600 (AbstractChannelHandlerContext.java:61)
at io.netty.channel.AbstractChannelHandlerContext$7.run (AbstractChannelHandlerContext.java:370)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute (AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks (SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run (NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run (SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run (ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run (FastThreadLocalRunnable.java:30)

Upgrade Netty to 4.1.54.Final

Other

Client: URL encode / escaping is wrong

Wrong pool is used to release `IOBuffer` after `ByteChannelSequential.copyTo` from static initialized instance.

`ByteBufferChannel.readRemaining` doesn't read whole channel

Update kotlin to 1.4.20

Fix configuration if project without VPN and cache

Handle failure in reading request body

Remove copyTo usage from ServerPipeline

1.4.2

released 10th November 2020

Client

Ktor Client HTTP Cache for invalid or blank Expires header

https://youtrack.jetbrains.com/issue/KTOR-364

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1892

Ktor Version and Engine Used (client or server and name) 1.3.2, client, io.ktor.client.features.cache.HttpCache

Describe the bug If a server returns a blank or an invalid Expires HTTP header, the ktor client crashes if the feature HttpCache is enabled install(HttpCache)

To Reproduce Steps to reproduce the behavior:

Write the following:

val articleUrl = Url("https://www.nachrichten.at/panorama/weltspiegel/30-jahre-mauerfall-sinnbild-der-freiheit;art17,3184503")
val newsClient = HttpClient(Apache) {
    install(HttpCache)

    install(UserAgent) {
        agent = "Mozilla/5.0"
    }
}
try {
    val call: HttpStatement = newsClient.get(articleSimilarityRequest.articleUrl)
    val response: HttpResponse = call.receive()
    val requestUrl = response.request.url.toString()
    val rawHtml = response.readText()
    println(rawHtml)
} catch (cause: Throwable) {
    println("Unable to download article")
} finally {
    newsClient.close()
}

If you open the URL https://www.nachrichten.at/panorama/weltspiegel/30-jahre-mauerfall-sinnbild-der-freiheit;art17,3184503 in the browser, you can see that the Expires response header is blank
An exception is thrown if you run the code:

2020-05-23 14:41:24.124 [DefaultDispatcher-worker-2 @request#48] ERROR a.s.api.util.service.DownloadService - Unable to download article
java.lang.IllegalStateException: Failed to parse date: 
	at io.ktor.http.DateUtilsKt.fromHttpToGmtDate(DateUtils.kt:40)
	at io.ktor.client.features.cache.HttpCacheEntryKt.cacheExpires(HttpCacheEntry.kt:84)
	at io.ktor.client.features.cache.HttpCacheEntryKt.HttpCacheEntry(HttpCacheEntry.kt:18)
	at io.ktor.client.features.cache.HttpCacheEntryKt$HttpCacheEntry$1.invokeSuspend(HttpCacheEntry.kt)
	(Coroutine boundary)
	at io.ktor.client.HttpClient$1.invokeSuspend(HttpClient.kt:123)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:68)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:116)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:79)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:89)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:34)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:164)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:43)
	at app.shwoo.api.util.service.DownloadService.downloadArticle$suspendImpl(DownloadService.kt:101)
	at app.shwoo.api.server.web.SimilarityResourceKt$similarity$1$4.invokeSuspend(SimilarityResource.kt:38)
	at io.ktor.routing.Routing.executeResult(Routing.kt:147)
	at io.ktor.routing.Routing.interceptor(Routing.kt:34)
	at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:99)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:107)
	at io.ktor.server.testing.TestApplicationEngine$callInterceptor$1.invokeSuspend(TestApplicationEngine.kt:292)
	at io.ktor.server.testing.TestApplicationEngine$2.invokeSuspend(TestApplicationEngine.kt:50)
	at io.ktor.server.testing.TestApplicationEngine$handleRequest$pipelineJob$1.invokeSuspend(TestApplicationEngine.kt:290)
Caused by: java.lang.IllegalStateException: Failed to parse date: 
	at io.ktor.http.DateUtilsKt.fromHttpToGmtDate(DateUtils.kt:40)
	at io.ktor.client.features.cache.HttpCacheEntryKt.cacheExpires(HttpCacheEntry.kt:84)
	at io.ktor.client.features.cache.HttpCacheEntryKt.HttpCacheEntry(HttpCacheEntry.kt:18)
	at io.ktor.client.features.cache.HttpCacheEntryKt$HttpCacheEntry$1.invokeSuspend(HttpCacheEntry.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Expected behavior It seems that the problem occurs in https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-core/common/src/io/ktor/client/features/cache/HttpCacheEntry.kt#L84. If the Expires header is blank an exception is thrown. The expected behavior is that the application handles the cache as expired if an invalid date is provided in the HTTP Expires header. Like described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expires

Content-type header is missing in Request when using LogLevel.BODY

NoTransformationFoundException when using JsonFeature and connecting through a proxy server

https://youtrack.jetbrains.com/issue/KTOR-955

Steps of reproduction:

Add windows or macOS targets to the Gradle build file if required.
Run ./gradlew runDebugExecutableLinuxX64 (replace with your target)

As a result, the following exception is thrown:

> Task :runDebugExecutableLinuxX64 FAILED
Uncaught Kotlin exception: io.ktor.client.call.NoTransformationFoundException: No transformation found: class io.ktor.utils.io.ByteChannelNative -> class Mine
with response from https://httpbin.org/get:
status: 200 OK
response headers: 

    at kfun:kotlin.Throwable#<init>(){} (0x306ebe)
    at kfun:kotlin.Exception#<init>(){} (0x3007b7)
    at kfun:kotlin.RuntimeException#<init>(){} (0x300377)
    at kfun:kotlin.UnsupportedOperationException#<init>(){} (0x300bf7)
    at kfun:io.ktor.client.call.NoTransformationFoundException#<init>(io.ktor.client.statement.HttpResponse;kotlin.reflect.KClass<*>;kotlin.reflect.KClass<*>){} (0x579a53)
    at kfun:io.ktor.client.call.HttpClientCall.$receiveCOROUTINE$12#invokeSuspend(kotlin.Result<kotlin.Any?>){}kotlin.Any? (0x578b21)
    at kfun:io.ktor.client.call.HttpClientCall#receive(io.ktor.client.call.TypeInfo){}kotlin.Any (0x578fc6)
    at kfun:$main$lambda-2COROUTINE$0.invokeSuspend#internal (0x640c1f)
    at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x322d4f)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
    at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
Unfinished workers detected, 1 workers leaked!
Use `Platform.isMemoryLeakCheckerActive = false` to avoid this check.
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
    at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
    at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x527128)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x526c9f)
    at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x528c6c)
    at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} (0x323007)
    at kfun:kotlinx.coroutines.DispatchedTask#run(){} (0x4114a6)
    at kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long (0x3e9ace)
    at kfun:kotlinx.coroutines#runEventLoop(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>){} (0x423639)
    at kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal (0x422de0)
    at kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 (0x42223b)
    at kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 (0x422741)
    at kfun:#main(){} (0x63e8cc)
    at Konan_start (0x641839)
    at Init_and_run_start (0x66cdcb)
    at __libc_start_main (0x7f0318cb20b3)
    at  (0x2b8029)
    at  ((nil))

Here is the related Github issue.

"IllegalStateException: Cannot execute task because event loop was shut down" for native client

https://youtrack.jetbrains.com/issue/KTOR-1071

To reproduce run ./gradlew runDebugExecutableMacosX64 in the root directory of the attached project. As a result, I get the following unexpected exception:

Uncaught Kotlin exception: kotlin.IllegalStateException: Cannot execute task because event loop was shut down
    at 0   loop_shutdown_coroutines.kexe       0x000000010b88270d kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23:37)
    at 1   loop_shutdown_coroutines.kexe       0x000000010b87ba7b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
    at 2   loop_shutdown_coroutines.kexe       0x000000010b87bc3b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
    at 3   loop_shutdown_coroutines.kexe       0x000000010b87c14b kfun:kotlin.IllegalStateException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:70:44)
    at 4   loop_shutdown_coroutines.kexe       0x000000010ba5639e kfun:kotlinx.coroutines#loopWasShutDown(){}kotlin.Nothing + 222 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/CoroutineContext.kt:25:50)
    at 5   loop_shutdown_coroutines.kexe       0x000000010ba5887d kfun:kotlinx.coroutines.EventLoopImplPlatform#reschedule(kotlin.Long;kotlinx.coroutines.EventLoopImplBase.DelayedTask){} + 61 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/EventLoop.kt:22:9)
    at 6   loop_shutdown_coroutines.kexe       0x000000010b9e657b kfun:kotlinx.coroutines.EventLoopImplBase.rescheduleAllDelayed#internal + 587 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:397:13)
    at 7   loop_shutdown_coroutines.kexe       0x000000010b9e2dde kfun:kotlinx.coroutines.EventLoopImplBase#shutdown(){} + 270 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:226:9)
    at 8   loop_shutdown_coroutines.kexe       0x000000010ba58d1c kfun:kotlinx.coroutines.EventLoopImpl#shutdown(){} + 140 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/EventLoop.kt:34:15)
    at 9   loop_shutdown_coroutines.kexe       0x000000010b9e12a4 kfun:kotlinx.coroutines.EventLoop#decrementUseCount(kotlin.Boolean){} + 324 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:114:13)
    at 10  loop_shutdown_coroutines.kexe       0x000000010b9e137c kfun:kotlinx.coroutines.EventLoop#decrementUseCount$default(kotlin.Boolean;kotlin.Int){} + 124 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:108:5)
    at 11  loop_shutdown_coroutines.kexe       0x000000010ba5383b kfun:kotlinx.coroutines#runEventLoop(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>){} + 1115 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:85:20)
    at 12  loop_shutdown_coroutines.kexe       0x000000010ba551f0 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 464 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:67:9)
    at 13  loop_shutdown_coroutines.kexe       0x000000010ba52dcf kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 1311 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:50:22)
    at 14  loop_shutdown_coroutines.kexe       0x000000010ba5330c kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 348 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/Builders.kt:33:15)
    at 15  loop_shutdown_coroutines.kexe       0x000000010b82bc48 kfun:#main(){} + 168 (/Users/Aleksei.Tirman/projects/jb/loop_shutdown_coroutines/src/macosX64Main/kotlin/App.kt:6:14)
    at 16  loop_shutdown_coroutines.kexe       0x000000010b82c5c0 Konan_start + 144 (/Users/Aleksei.Tirman/projects/jb/loop_shutdown_coroutines/src/macosX64Main/kotlin/App.kt:6:1)
    at 17  loop_shutdown_coroutines.kexe       0x000000010b831f4b Init_and_run_start + 107
    at 18  libdyld.dylib                       0x00007fff6a1c9cc9 start + 1
Unfinished workers detected, 1 workers leaked!
Use `Platform.isMemoryLeakCheckerActive = false` to avoid this check.
Abort trap: 6

Here is the similar issue. I've reproduced this problem for both macOS and Linux targets. Kotlin multiplatform: 1.4.10 Kotlin coroutines: 1.3.9-native-mt-2 Ktor: 1.4.0

Native: ResponseObserver hangs coroutine indefinitely

Client: InvalidCacheStateException when Vary header differs for 200 and 304 responses

IncorrectDereferenceException thrown when ktor request call from other thread

Jetty client engine with high concurrency randomly freezes, throws NPEs, ISEs or EofExceptions

https://youtrack.jetbrains.com/issue/KTOR-1120

Exception in thread "main" java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:222)
	at org.eclipse.jetty.io.SelectorManager.connect(SelectorManager.java:193)
	at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:409)
	at org.eclipse.jetty.http2.client.HTTP2Client.connect(HTTP2Client.java:382)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt:61)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$connect$2.invoke(JettyHttpRequest.kt)
	at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt.connect(JettyHttpRequest.kt:59)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:31)
	at io.ktor.client.engine.jetty.JettyHttp2Engine.execute(JettyHttp2Engine.kt:38)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
	Suppressed: java.lang.NullPointerException
		... 17 more
	Suppressed: java.lang.NullPointerException
		... 17 more
	Suppressed: java.lang.NullPointerException
		... 17 more
	Suppressed: java.lang.NullPointerException
		... 17 more

import io.ktor.client.*
import io.ktor.client.engine.jetty.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.*

suspend fun main() = withContext(Dispatchers.Default) {
   HttpClient(Jetty) {
      expectSuccess = false

      install(HttpTimeout) {
         connectTimeoutMillis = 2_000
         socketTimeoutMillis = 2_000
      }

      engine {
         threadsCount = 200
      }
   }.use { client ->
      coroutineScope {
         repeat(5) {
            repeat(200) {
               launch {
                  client.get<HttpResponse>("https://esi.evetech.net/latest/markets/10000040/orders/?page=1")
                     .readText()
                     .let { println(it.length) }
               }
            }
         }
      }
   }

   println()
   println("Done")
}

Ktor 1.4.1 fails to upload file (submit form) when logging level = LogLevel.BODY

CertificatePinner buildErrorMessage outputs CertificatesInfo reference and variable name (HASH_ALGORITHM_SHA_256) instead of value

Overload resolution ambiguity: HttpClient.ws

Core

CIO client engine randomly runs into infinite loop under high load

https://youtrack.jetbrains.com/issue/KTOR-1126

Stuck in while (!closed) in io.ktor.network.selector.ActorSelectorManager.process(). Probably related to KTOR-717. Likely a duplicate of KTOR-531.

I also randomly get this:

Exception in thread "main" java.net.BindException: Can't assign requested address
	at java.base/sun.nio.ch.Net.connect0(Native Method)
	at java.base/sun.nio.ch.Net.connect(Net.java:503)
	at java.base/sun.nio.ch.Net.connect(Net.java:492)
	at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:751)
	at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:31)
	at io.ktor.network.sockets.ConnectUtilsJvmKt.connect(ConnectUtilsJvm.kt:19)
	at io.ktor.network.sockets.TcpSocketBuilder.connect(TcpSocketBuilder.kt:38)
	at io.ktor.client.engine.cio.ConnectionFactory.connect(ConnectionFactory.kt:24)
	at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invokeSuspend(Endpoint.kt:159)
	at io.ktor.client.engine.cio.Endpoint$connect$$inlined$repeat$lambda$1.invoke(Endpoint.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:120)
	at kotlinx.coroutines.TimeoutKt.withTimeoutOrNull(Timeout.kt:83)
	at io.ktor.client.engine.cio.Endpoint.connect(Endpoint.kt:167)
	at io.ktor.client.engine.cio.Endpoint$makeDedicatedRequest$1.invokeSuspend(Endpoint.kt:95)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more
	Suppressed: java.net.BindException: Can't assign requested address
		... 21 more

Repro:

import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.cio.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import kotlin.random.*
import kotlin.time.*
import kotlinx.coroutines.*

@OptIn(ExperimentalTime::class)
suspend fun main() = withContext(Dispatchers.Default) {
    startServer()
    delay(1_000)

    val batchSize = 100
    val threadCount = 1

    HttpClient(CIO) {
        engine {
            threadsCount = threadCount
        }
    }.use { client ->
        repeat(Int.MAX_VALUE) { i ->
            val time = measureTime {
                coroutineScope {
                    repeat(batchSize) {
                        launch {
                            client.get<HttpResponse>("http://localhost:8080").readText()
                        }
                    }
                }
            }

            println("${i + 1}: $time")
        }
    }
}

private fun startServer() {
    embeddedServer(Netty, port = 8080) {
        val random = Random(0)
        val data = buildString(100_000) { repeat(100_000) { append(random.nextInt(0, 10)) } }

        routing {
            get("/") {
                call.respondText(data, ContentType.Text.Plain)
            }
        }
    }.start()
}

Infrastructure

ktor-server-test-host indirectly depends on logback-classic

Add artifact dependency to release configuration

Plugin

Test compilation errors when project created via plugin with "Mock HttpClient Engine"

No changelog for 1.4.0 and 1.4.1 on the plugin page in IDE Marketplace

Plugin: the Ktor project type tooltip in the New Project wizard contains text used for Java projects

Plugin: "Could not find io.ktor:ktor-client-http-timeout" when building project created with http-timeout feature

Server

Class cast exception on web-socket cancelation

https://youtrack.jetbrains.com/issue/KTOR-349

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1929

Ktor Version and Engine Used (client or server and name) ktor-cio server 1.3.2

Describe the bug I am getting an internal error when force-closing server with opened web-socket connection Here is the stack trace:

13:32:20.839 [DefaultDispatcher-worker-4] ERROR io.ktor.server.cio.HttpServer - Unhandled exception caught for CoroutineName(http-pipeline-writer)
kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[ExperimentalCoroutineDispatcher@45e6d6a4[scheduler = DefaultDispatcher@6341301e[Pool Size {core = 10, max = 1024}, Worker States {CPU = 6, blocking = 0, parked = 8, dormant = 0, terminated = 0}, running workers queues = [0c, 0c, 0c, 0c, 0c, 0c], global CPU queue size = 5, global blocking queue size = 0, Control State {created workers= 14, blocking tasks = 0, CPUs acquired = 6}]], Continuation at io.ktor.server.cio.backend.ServerPipelineKt$startServerConnectionPipeline$1$outputsActor$1.invokeSuspend(ServerPipeline.kt:49)@2482cf33]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
	at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
	at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
	at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
	... 4 common frames omitted

CORS - request with non-simple content type not failing

Scheme instead of version in OriginConnectionPoint

Setting multiple cache control directives is impossible with current API

Sensible defaults for compression feature

Netty: NumberFormatException when make a request via HTTP/2 without explicit port

https://youtrack.jetbrains.com/issue/KTOR-1091

Hello.

There is a bug with Http2LocalConnectionPoint. If there is no port specified in authority then it tries to parse domain as port. I've created a project to reproduce the bug: https://github.com/Vlad8161/netty-bug-sample

To reproduce it run the server on 443 port and access it without explicit port specifying.

Stack Trace:

java.lang.NumberFormatException: For input string: "localhost"
     	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
 	at java.lang.Integer.parseInt(Integer.java:580)
 	at java.lang.Integer.parseInt(Integer.java:615)
 	at io.ktor.server.netty.http2.Http2LocalConnectionPoint.getPort(Http2LocalConnectionPoint.kt:31)
 	at io.ktor.features.HSTS.intercept(HSTS.kt:87)
 	at io.ktor.features.HSTS$Feature$install$1.invokeSuspend(HSTS.kt:101)
 	at io.ktor.features.HSTS$Feature$install$1.invoke(HSTS.kt)
 	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
 	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
 	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:101)
 	at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt)
 	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91)
 	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:189)
 	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:100)
 	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:140)
 	at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt)
 	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
 	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
 	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139)
 	at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt)
 	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
 	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
 	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
 	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
 	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
 	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
 	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
 	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
 	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
 	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
 	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
 	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
 	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
 	at kotlinx.coroutines.BuildersKt__Builders_commonKt.startCoroutineImpl(Builders.common.kt:182)
 	at kotlinx.coroutines.BuildersKt.startCoroutineImpl(Unknown Source)
 	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:145)
 	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:54)
 	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
 	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
 	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
 	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 	at java.lang.Thread.run(Thread.java:748)

The problem is at Http2LocalConnectionPoint:31

I think is would be better to do something like this:

nettyHeaders.authority()?.toString()?.substringAfter(":", if (scheme == "https") "443" else "80")?.toInt()

because often local port information is not enough

Thanks in advance :)

Make Jackson serialization stream response

Test Infrastructure

Replace JUnit with kotlin.test

Other

TestHttpClientEngine KNPE

Ktor Client Null Pointer Exception (HTTPS)

Ktor CIO client, connect timeout handling

https://youtrack.jetbrains.com/issue/KTOR-446

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1720

Using the following code (full example at https://github.com/rrva/ktor-client-connection-leak)
Run the main method repeatedly. Sometimes, all futures created will not complete . Why? Expected behavior is that all created futures will complete eventually, so Done with all will be printed.

 class Client(val fooServiceUrl: String) {

    val client = HttpClient(CIO) {
        engine {
            requestTimeout = 500
            endpoint.connectTimeout = 100
            endpoint.connectRetryAttempts = 2
        }
        install(JsonFeature) {
            serializer = KotlinxSerializer(
                Json(
                    JsonConfiguration(
                        isLenient = true,
                        ignoreUnknownKeys = true,
                        serializeSpecialFloatingPointValues = true,
                        useArrayPolymorphism = true
                    )
                )
            )
        }
    }

    suspend fun fetchFoo(ids: List<String>): List<Foo> {
            return client.request {
                header("User-Agent", "Foo")
                parameter("input", ids.joinToString(","))
                url("$fooServiceUrl/foo")
            }
    }

}

@Serializable
data class Foo(val id: String)
val client = Client("http://localhost:9090")

fun fetchFooAsync(ids:List<String>): CompletableFuture<List<Foo>> {
    return CoroutineScope(Job() + Dispatchers.IO + MDCContext()).async {
        try {
            client.fetchFoo(ids)
        } catch (e: Throwable) {
            println(e.message)
            listOf<Foo>(Foo("timeout"))
        }
    }.asCompletableFuture()
}


fun main() {
    HttpServer.create(InetSocketAddress(9090), 1000).apply {

        createContext("/foo") { http ->
            http.responseHeaders.add("Content-type", "application/json")
            http.sendResponseHeaders(200, 0)
            PrintWriter(http.responseBody).use { out ->
                out.println("""[{"id":"1"}]""")
            }
        }

        start()
    }
    println("Start")

    val requests = (1..100).map {
        fetchFooAsync(listOf(it.toString()))
    }
    for (request in requests) {
        println("Waiting for coroutine...")
        println(request.get())
        println("Done waiting")
    }

    println("Done with all")

}

Reproduced on 1.3.1

Url(…) doesn't parse non-HTTP-like URL strings correctly

"response.readText()" never returns (probably infinite loop)

Create sample with native CIO client and `native-mt` coroutines

Split ktor-server-test-host module

Extract flaky tests from `master` to the `flaky-tests` branch

Blocking calls (ReentrantLock.lock) on Apache engine dispatcher threads

https://youtrack.jetbrains.com/issue/KTOR-1140

I'm using BlockHound to check if there are any blocking calls within my coroutines that aren't using IO dispatcher. BlockHound reports the following occasion for Ktor client with Apache engine on thread ktor-apache-dispatcher-worker-4. Maybe it's a bug, maybe it's a misconfiguration of the dispatcher threads as CPU-bound.

reactor.blockhound.BlockingOperationError: Blocking call! jdk.internal.misc.Unsafe#park
	at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
	at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
	at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:68)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:33)
	at io.ktor.client.engine.apache.ApacheEngine.execute(ApacheEngine.kt:37)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Fix HTTP2 tests with new versions of java8

1.4.1

released 24th September 2020

Client

How to use ContentEncoding?

Parser Exception in header with character code 1 not allowed

https://youtrack.jetbrains.com/issue/KTOR-860

The following URL

https://data.smartdublin.ie/cgi-bin/rtpi

returns valid results with other HTTP clients, including IntelliJ HTTP Requests. However with Ktor, independently of engine, it throws a parser exception (side note, it would be good to indicate the header in the exception result)

Exception in thread "main" io.ktor.http.cio.ParserException: Character with code 1 is not allowed in header names, 
HTTP/1.1 200 OKServer: nginx/1.17.8Date: Sun, 19 Jul 2020 07:12:42 GMTContent-Type: application/json; charset=utf-8Content-Length: 1596054Connection: keep-aliveX-Powered-By: ASP.NETSet-Cookie: visid_incap_1822411=6ycr4UzsSv+6sdm+QnNmpmnyE18AAAAAQUIPAAAAAACm9JYEnYu4a0fTA7VxKyxu; expires=Sun, 18 Jul 2021 08:40:35 GMT; HttpOnly; path=/; Domain=.nationaltransport.ieSet-Cookie: incap_ses_535_1822411=vjDoPwoqvTdguklauLNsB2nyE18AAAAAoieq1zxaD46bWhzEJyQd3w==; path=/; Domain=.nationaltransport.ieSet-Cookie: ___utmvmzauvysSB=WQYQExBjiZv; path=/; Max-Age=900Set-Cookie: ___utmvazauvysSB=TjhnJRz; path=/; Max-Age=900
	at io.ktor.http.cio.HttpParserKt.characterIsNotAllowed(HttpParser.kt:279)
	at io.ktor.http.cio.HttpParserKt.parseHeaderValue(HttpParser.kt:261)
	at io.ktor.http.cio.HttpParserKt.parseHeaders(HttpParser.kt:119)
	at io.ktor.http.cio.HttpParserKt.parseResponse(HttpParser.kt:73)
	at io.ktor.http.cio.HttpParserKt$parseResponse$1.invokeSuspend(HttpParser.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Process finished with exit code 1

To reproduce see attached code.

Native: InvalidMutabilityException creating HttpClient

https://youtrack.jetbrains.com/issue/KTOR-915

As of v.1.4.0, ktor client is not working on Kotlin/native iOS due to an InvalidMutabilityException.

I tested with both coroutines "1.3.9" as well as "1.3.9-native-mt", each produce the same error:

Uncaught Kotlin exception: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen kotlinx.coroutines.ChildHandleNode@3873b88
    at 0   jgosdk                              0x000000010b7376fd kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Throwable.kt:23:37)
    at 1   jgosdk                              0x000000010b73030b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44)
    at 2   jgosdk                              0x000000010b73055b kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44)
    at 3   jgosdk                              0x000000010b767deb kfun:kotlin.native.concurrent.InvalidMutabilityException#<init>(kotlin.String){} + 91 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Freezing.kt:22:60)
    at 4   jgosdk                              0x000000010b7682c2 ThrowInvalidMutabilityException + 690 (/Users/teamcity/buildAgent/work/cae0e6559deed4c4/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:92:11)
    at 5   jgosdk                              0x000000010b85e9ec MutationCheck + 108
    at 6   jgosdk                              0x000000010b9017b6 kfun:kotlinx.coroutines.internal.LinkedListNode#<set-_next>(kotlinx.coroutines.internal.LinkedListNode){} + 102 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:19:28)
    at 7   jgosdk                              0x000000010b901d9b kfun:kotlinx.coroutines.internal.LinkedListNode#addLast(kotlinx.coroutines.internal.LinkedListNode){} + 283 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:31:9)
    at 8   jgosdk                              0x000000010b9020e6 kfun:kotlinx.coroutines.internal.LinkedListNode#addOneIfEmpty(kotlinx.coroutines.internal.LinkedListNode){}kotlin.Boolean + 230 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/native/src/internal/LinkedList.kt:47:9)
    at 9   jgosdk                              0x000000010b8a3cbf kfun:kotlinx.coroutines.JobSupport.promoteSingleToNodeList#internal + 463 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:532:15)
    at 10  jgosdk                              0x000000010b8a291d kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Boolean;kotlin.Boolean;kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 2061 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:470:25)
    at 11  jgosdk                              0x000000010b8a208f kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 207 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/JobSupport.kt:449:9)
    at 12  jgosdk                              0x000000010ba59d04 kfun:io.ktor.client#HttpClient(io.ktor.client.engine.HttpClientEngineFactory<0:0>;kotlin.Function1<io.ktor.client.HttpClientConfig<0:0>,kotlin.Unit>){0§<io.ktor.client.engine.HttpClientEngineConfig>}io.ktor.client.HttpClient + 1156 (/opt/buildAgent/work/a85294440dc5c6e/ktor-client/ktor-client-core/common/src/io/ktor/client/HttpClient.kt:45:36)

1.4.0: breaking change by making response nullable in ResponseException

Ktor websocket client passes configured max frame as timeout millis

CIO: java.io.EOFException when server requires client certificate and restricts the authority

	reading	remainingBefore	remainingAfter	content	position
before		281	281
reading authorizationSize	2	281	279	279
reading size	2	279	277	91
reading CA 1	91	277	186	CA...	91 (should 93)
reading size	2	186	184	91
reading CA 2	91	184	93	CA	182 (should 186)
reading size	2	93	91	91
reading CA 3	91	91	0	CA	273 (should 279)

Implement runtime check of using `native-mt` coroutines

"FreezingException: freezing of InvokeOnCompletion has failed" using native-mt coroutines

https://youtrack.jetbrains.com/issue/KTOR-973

Hi. I have a problem on Ktor client native. I’ve been struggling to make the code below works after migrating kotlin to 1.4.0 version while migrating, I changed coroutine version to 1.3.9-native-mt

class TestFreezingViewModel(val client: HttpClient) {
    fun call() {
         CoroutineScope(Dispatchers.Main).launch(start = CoroutineStart.LAZY) {
            try {
                client.get<String>("https://test.com") {}
            } catch (e: Throwable) {
                e.printStackTrace()
            }
        }.start()
    }
}

swift side code

TestFreezingViewModel().call()

while the code running, The error below occurs

kotlin.native.concurrent.FreezingException: freezing of InvokeOnCompletion[InvokeOnCompletion@1b76e08] has failed, first blocker is HttpClient[io.ktor.client.engine.ios.IosClientEngine@1b1b748]
    at 0   kotlinIOS                           0x000000010bf93fed kfun:kotlin.Throwable#<init>(kotlin.String?){} + 93
    at 1   kotlinIOS                           0x000000010bf8cb9b kfun:kotlin.Exception#<init>(kotlin.String?){} + 91
    at 2   kotlinIOS                           0x000000010bf8cdeb kfun:kotlin.RuntimeException#<init>(kotlin.String?){} + 91
    at 3   kotlinIOS                           0x000000010bfc2c13 kfun:kotlin.native.concurrent.FreezingException#<init>(kotlin.Any;kotlin.Any){} + 643
    at 4   kotlinIOS                           0x000000010bfc4297 ThrowFreezingException + 231
    at 5   kotlinIOS                           0x000000010c0b2f4f FreezeSubgraph + 2815
    at 6   kotlinIOS                           0x000000010c0cc17b Kotlin_Worker_freezeInternal + 27
    at 7   kotlinIOS                           0x000000010bfc2d3e kfun:kotlin.native.concurrent#freeze@0:0(){0§<kotlin.Any?>}0:0 + 62
    at 8   kotlinIOS                           0x000000010c10c749 kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Boolean;kotlin.Boolean;kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 1881
    at 9   kotlinIOS                           0x000000010c10bf76 kfun:kotlinx.coroutines.JobSupport#invokeOnCompletion(kotlin.Function1<kotlin.Throwable?,kotlin.Unit>){}kotlinx.coroutines.DisposableHandle + 198
    at 10  kotlinIOS                           0x000000010c33b15b kfun:io.ktor.client.engine#createCallContext@io.ktor.client.engine.HttpClientEngine(kotlinx.coroutines.Job){}kotlin.coroutines.CoroutineContext + 1451
    at 11  kotlinIOS                           0x000000010c302d08 kfun:io.ktor.client.engine.HttpClientEngine.$executeWithinCallContextCOROUTINE$9.invokeSuspend#internal + 1048
    at 12  kotlinIOS                           0x000000010c303444 kfun:io.ktor.client.engine.HttpClientEngine.executeWithinCallContext#internal + 308
    at 13  kotlinIOS                           0x000000010c30420a kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invokeSuspend#internal + 1562
    at 14  kotlinIOS                           0x000000010c304bf8 kfun:io.ktor.client.engine.HttpClientEngine.$install$lambda-0COROUTINE$10.invoke#internal + 312
    at 15  kotlinIOS                           0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
    at 16  kotlinIOS                           0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395
    at 17  kotlinIOS                           0x000000010c29d50a kfun:io.ktor.util.pipeline.SuspendFunctionGun.execute#internal + 474
    at 18  kotlinIOS                           0x000000010c295759 kfun:io.ktor.util.pipeline.Pipeline#execute(1:1;1:0){}1:0 + 361
    at 19  kotlinIOS                           0x000000010c326be5 kfun:io.ktor.client.features.HttpSend.DefaultSender.$executeCOROUTINE$27.invokeSuspend#internal + 1365
    at 20  kotlinIOS                           0x000000010c327244 kfun:io.ktor.client.features.HttpSend.DefaultSender.execute#internal + 308
    at 21  kotlinIOS                           0x000000010c324fbc kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invokeSuspend#internal + 2236
    at 22  kotlinIOS                           0x000000010c3260f8 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0COROUTINE$26.invoke#internal + 312
    at 23  kotlinIOS                           0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
    at 24  kotlinIOS                           0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395
    at 25  kotlinIOS                           0x000000010c29d2a5 kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceedWith#internal + 213
    at 26  kotlinIOS                           0x000000010c3173eb kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-0COROUTINE$17.invokeSuspend#internal + 907
    at 27  kotlinIOS                           0x000000010c317c58 kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-0COROUTINE$17.invoke#internal + 312
    at 28  kotlinIOS                           0x000000010c29da5f kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 1135
    at 29  kotlinIOS                           0x000000010c29d0ab kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 395

when Coroutine’s invokeOnCompletion is invoked, freeze() is called. and it tells that HttpClient can’t be frozen When I checked the source code HttpClient.init calls preventFreeze() I would like to know the intention of preventFreeze and what is the proper approach to use on developer’s side. please check if this is bug, other wise, kindly let me share some advice. Thanks

Calling HttpStatement#toString more than once throws IllegalArgumentException

Core

ConcurrentList.increaseCapacity() throws ArrayIndexOutOfBoundsException

https://youtrack.jetbrains.com/issue/KTOR-1034

I have a test that uses MockEngine which makes more than 32 calls, and then throws a java.lang.ArrayIndexOutOfBoundsException (stacktrace at the bottom).

I tracked the problem down to io.ktor.util.collections.ConcurrentList.increaseCapacity() (ConcurrentList.kt:180):

    private fun increaseCapacity(targetCapacity: Int = data.size * 2) {
        val newData = SharedList<T>(targetCapacity)
        for (index in 0 until targetCapacity) {
            newData[index] = data[index]
        }

        data = newData
    }

It is clear from this code that the for loop uses targetCapacity which is going to try to access data outside of the original data size. Here is what the code should be:

    private fun increaseCapacity(targetCapacity: Int = data.size * 2) {
        val newData = SharedList<T>(targetCapacity)
        for (index in 0 until data.size) {
            newData[index] = data[index]
        }

        data = newData
    }

Here is the stacktrace for reference.

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 32 out of bounds for length 32
	at java.base/java.lang.invoke.VarHandle$1.apply(VarHandle.java:2011)
	at java.base/java.lang.invoke.VarHandle$1.apply(VarHandle.java:2008)
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:159)
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:156)
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:62)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.lang.invoke.VarHandleObjects$Array.getVolatile(VarHandleObjects.java:438)
	at java.base/java.lang.invoke.VarHandleGuards.guard_LI_L(VarHandleGuards.java:646)
	at java.base/java.util.concurrent.atomic.AtomicReferenceArray.get(AtomicReferenceArray.java:100)
	at io.ktor.util.collections.internal.SharedList.get(SharedList.kt:35)
	at io.ktor.util.collections.ConcurrentList.increaseCapacity(ConcurrentList.kt:180)
	at io.ktor.util.collections.ConcurrentList.increaseCapacity$default(ConcurrentList.kt:177)
	at io.ktor.util.collections.ConcurrentList.add(ConcurrentList.kt:65)
	at io.ktor.client.engine.mock.MockEngine.execute(MockEngine.kt:63)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:333)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.startCoroutineImpl(Builders.common.kt:188)
	at kotlinx.coroutines.BuildersKt.startCoroutineImpl(Unknown Source)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:145)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91)
	at kotlinx.coroutines.BuildersKt.async(Unknown Source)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84)
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:124)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:88)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:185)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at com.gatsby.trade.client.OrbisOrderClient$placeOrder$$inlined$post$1.invokeSuspend(OrbisClient.kt:242)

Upgrade kotlinx.serialization to 1.0.0-RC2

Infrastructure

Enable binary-compatibility-validator in all modules

Samples

Routing: get matcher has higher priority than param matcher on the same level

Server

Exception after WebSocketSession.close() invocation.

MultiPartData.readAllParts() throws java.io.IOException when multipart list is empty

https://youtrack.jetbrains.com/issue/KTOR-767

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/482

Problem description: When receiving an empty multipart-form request on a POST route, the request handler throws an IOException, preventing graceful handling or the display of an error message. Instead, a 500 error message is returned.

Minimum working example:

import io.ktor.application.*
import io.ktor.content.readAllParts
import io.ktor.response.*
import io.ktor.request.*
import io.ktor.routing.post
import io.ktor.routing.routing

fun main(args: Array<String>): Unit = io.ktor.server.netty.DevelopmentEngine.main(args)

fun Application.module() {
    routing {
        post("/add") {
            if(call.request.isMultipart()) {
                val multipart = call.receiveMultipart()
                val formItems = multipart.readAllParts() // <-- this breaks when sent an empty request
                call.respondText("recvd: $multipart")
            }
            else {
                call.respond("no multipart, fix request!")
            }
        }
    }
}

Send a request with an empty multipart body to see the issue:

curl -X POST \
  http://localhost:8080/add \
  -H 'Cache-Control: no-cache' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'

Error Message:

22:57:11.866 [nettyCallPool-4-4] ERROR Application - Unhandled: POST - /add
java.io.IOException: Broken delimiter occurred
	at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.doResume(Delimited.kt:57)
	at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.invoke(Delimited.kt)
	at kotlinx.coroutines.experimental.io.DelimitedKt$skipDelimiterSuspend$2.invoke(Delimited.kt)
	at kotlinx.coroutines.experimental.io.ByteBufferChannel.lookAheadSuspend(ByteBufferChannel.kt:1746)
	at kotlinx.coroutines.experimental.io.DelimitedKt.skipDelimiterSuspend(Delimited.kt:55)
	at kotlinx.coroutines.experimental.io.DelimitedKt.skipDelimiter(Delimited.kt:50)
	at io.ktor.http.cio.MultipartKt.boundary(Multipart.kt:89)
	at io.ktor.http.cio.MultipartKt$parseMultipart$1.doResume(Multipart.kt:158)
	at kotlin.coroutines.experimental.jvm.internal.CoroutineImpl.resume(CoroutineImpl.kt:42)
	at kotlinx.coroutines.experimental.DispatchedKt.resumeCancellable(Dispatched.kt:209)
	at kotlinx.coroutines.experimental.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:35)
	at kotlinx.coroutines.experimental.CoroutineStart.invoke(CoroutineStart.kt:111)
	at kotlinx.coroutines.experimental.AbstractCoroutine.start(AbstractCoroutine.kt:165)
	at kotlinx.coroutines.experimental.channels.ProduceKt.produce(Produce.kt:95)
	at kotlinx.coroutines.experimental.channels.ProduceKt.produce$default(Produce.kt:88)
	at io.ktor.http.cio.MultipartKt.parseMultipart(Multipart.kt:145)
	at io.ktor.http.cio.MultipartKt.parseMultipart(Multipart.kt:138)
	at io.ktor.http.cio.CIOMultipartDataBase.<init>(CIOMultipartData.kt:33)
	at io.ktor.http.cio.CIOMultipartDataBase.<init>(CIOMultipartData.kt:31)
	at io.ktor.server.engine.DefaultTransformKt.multiPartData(DefaultTransform.kt:70)
	at io.ktor.server.engine.DefaultTransformKt.access$multiPartData(DefaultTransform.kt:1)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.doResume(DefaultTransform.kt:33)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.invoke(DefaultTransform.kt)
	at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$2.invoke(DefaultTransform.kt)
	at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
	at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
	at io.ktor.request.ApplicationReceiveFunctionsKt.receive(ApplicationReceiveFunctions.kt:64)
	at io.sebi.ApplicationKt$module$1$1.doResume(application.kt:31)
	at io.sebi.ApplicationKt$module$1$1.invoke(application.kt)
	at io.sebi.ApplicationKt$module$1$1.invoke(application.kt)
	at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
	at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
	at io.ktor.routing.Routing.executeResult(Routing.kt:100)
	at io.ktor.routing.Routing.interceptor(Routing.kt:25)
	at io.ktor.routing.Routing$Feature$install$1.doResume(Routing.kt:66)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt:51)
	at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
	at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.doResume(DefaultEnginePipeline.kt:66)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.pipeline.PipelineContext.proceed(PipelineContext.kt:49)
	at io.ktor.pipeline.Pipeline.execute(Pipeline.kt:22)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.doResume(NettyApplicationCallHandler.kt:31)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt:10)
	at kotlinx.coroutines.experimental.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:44)
	at kotlinx.coroutines.experimental.CoroutineStart.invoke(CoroutineStart.kt:113)
	at kotlinx.coroutines.experimental.AbstractCoroutine.start(AbstractCoroutine.kt:165)
	at kotlinx.coroutines.experimental.BuildersKt__Builders_commonKt.launch(Builders.common.kt:72)
	at kotlinx.coroutines.experimental.BuildersKt.launch(Unknown Source)
	at kotlinx.coroutines.experimental.BuildersKt__Builders_commonKt.launch$default(Builders.common.kt:64)
	at kotlinx.coroutines.experimental.BuildersKt.launch$default(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:22)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:16)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:844)

Confusing log message about failed session lookup

Netty: Not started servers leak resources

https://youtrack.jetbrains.com/issue/KTOR-939

Netty.create() does some heavy initialization immediately, for example creating EventLoopGroups. These in turn create pipes. Even though the engine is never started it allocates resources and would only release them when calling stop().

This was quite confusing in unit testing where I create engines repeatedly. They may or may not get started and only get stopped if they were actually started. I'd expect resources to be only allocated when I call start(), esp. since only stop() will release them.

repeat(1000) {
    embeddedServer(Netty, applicationEngineEnvironment { })
}

Exception in thread "main" java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:96)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:91)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:52)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion.create(NettyApplicationEngine.kt:190)
	at io.ktor.server.netty.NettyApplicationEngine.<init>(NettyApplicationEngine.kt:90)
	at io.ktor.server.netty.Netty.create(Embedded.kt:14)
	at io.ktor.server.netty.Netty.create(Embedded.kt:12)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:79)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:77)
	at server.MainKt.main(Main.kt:15)
	at server.MainKt$$$main.invoke(Unknown Source)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt$createCoroutineUnintercepted$$inlined$createCoroutineFromSuspendFunction$IntrinsicsKt__IntrinsicsJvmKt$1.invokeSuspend(IntrinsicsJvm.kt:205)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:115)
	at kotlin.coroutines.jvm.internal.RunSuspendKt.runSuspend(RunSuspend.kt:19)
	at server.MainKt.main(Main.kt)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 19 more
Caused by: java.io.IOException: Too many open files
	at java.base/sun.nio.ch.IOUtil.makePipe(Native Method)
	at java.base/sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:86)
	at java.base/sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:35)
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
	... 23 more

Wrong session id get stuck at clients

Other

Ktor 1.3.1 Fails File Upload with MalformedInputException

https://youtrack.jetbrains.com/issue/KTOR-391

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1845

Hi. I am using Ktor Client with version 1.3.1 for MPP

I have been using Ktor version 1.2.6 in my MPP project with Kotlin version 1.3.61 and it was working fine, so I was able to upload files without a problem. Later I update Kotlin version to 1.3.72 which forced me to update Ktor version 1.3.1 as version 1.2.6 was compiled with Kotlin version 1.3.61 which IDE can not compile with Kotlin plugin 1.3.72. After that file upload call started to give me the following exception.

2020-05-02 17:35:35.186 14564-14671/com.inivo.client.debug E/AndroidRuntime: FATAL EXCEPTION: ktor-android-dispatcher-worker-2
    Process: com.inivo.client.debug, PID: 14564
    io.ktor.utils.io.charsets.MalformedInputException: Input length = 1
        at io.ktor.utils.io.charsets.CharsetJVMKt.throwExceptionWrapped(CharsetJVM.kt:323)
        at io.ktor.utils.io.charsets.CharsetJVMKt.decode(CharsetJVM.kt:199)
        at io.ktor.utils.io.charsets.EncodingKt.decode(Encoding.kt:101)
        at io.ktor.utils.io.core.StringsKt.readText(Strings.kt:251)
        at io.ktor.utils.io.core.StringsKt.readText$default(Strings.kt:250)
        at io.ktor.client.features.logging.Logging$logRequestBody$2$1.invokeSuspend(Logging.kt:111)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoop.processUnconfinedEvent(EventLoop.common.kt:69)
        at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:184)
        at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
        at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:307)
        at kotlinx.coroutines.CancellableContinuationImpl.resumeImpl(CancellableContinuationImpl.kt:317)
        at kotlinx.coroutines.CancellableContinuationImpl.resumeWith(CancellableContinuationImpl.kt:249)
        at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2210)
        at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:930)
        at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:544)
        at io.ktor.utils.io.ByteBufferChannel.readAvailable$suspendImpl(ByteBufferChannel.kt:606)
        at io.ktor.utils.io.ByteBufferChannel.readAvailable(Unknown Source:0)
        at io.ktor.utils.io.jvm.javaio.WritingKt.copyTo(Writing.kt:21)
        at io.ktor.utils.io.jvm.javaio.WritingKt.copyTo$default(Writing.kt:12)
        at io.ktor.client.engine.android.AndroidClientEngineKt.writeTo(AndroidClientEngine.kt:112)
        at io.ktor.client.engine.android.AndroidClientEngine.execute(AndroidClientEngine.kt:79)
        at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:83)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

I am uploading a file like this.

override suspend fun sendImageAsync(
        token: String,
        file: File,
        fileType: String,
        id: Long
    ): Unit = client.request {
        fullUrl("${RestApi.SEND_IMAGE}/${id}")
        method = HttpMethod.Post
        body = MultiPartFormDataContent(
            formData {
                append(
                    "image",
                    file.fileData.bytes,
                    headers = Headers.build {
                        append(HttpHeaders.ContentType, "image/jpg")
                        append(
                            HttpHeaders.ContentDisposition,
                            "filename=${file.name}"
                        )
                        append("Connection", "Close")
                    }
                )
            }
        )
        header(HttpHeaders.Connection, "close")
        header(HttpHeaders.Authorization, makeAuthToken(token))
    }

Here file data is just an object which keeps File data as a byte array so by that I can convert image to byte array (either from Android / IOS) and upload it.

Thank you very much for your help in advanced.

MalformedInputException: Input length = 1, when uploading an image multipart

https://youtrack.jetbrains.com/issue/KTOR-425

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1766

Ktor Version and Engine Used (client or server and name) Ktor 1.3.2, Kotlin 1.3.70, Netty server, ContentNegotiation, Jackson parser, Exposed.

I simply used the Ktor project generator for Gradle, and checked a few boxes for modules as mentioned above. After which I managed to connect a database & a few endpoints, but my core endpoint for uploading images doesn't work.

Describe the bug When trying to upload a simple image using Retrofit & Multipart support in Android, I get a MalformedInputException on my server.

Retrofit/Android code:

@Multipart
@POST("/api/upload")
suspend fun uploadImages(@Part("image") imageFile: RequestBody): UploadResponse

Upload code:

val requestBody: RequestBody = MultipartBody.Builder()
        .setType(MultipartBody.FORM)
        .addFormDataPart("title", "Image Upload")
        .addFormDataPart(
            "image", // part name
            file.path.split("/").last(), // file name
            file.asRequestBody("image/jpg".toMediaType()) // Content-Type: image/jpg
        )
        .build()

    return apiService.uploadImages(requestBody)

Using the new way of creating RequestBody objects in OkHttp. I followed the documentation to receive the multipart data, and I've formed a MalformedInputException.

To Reproduce Steps to reproduce the behavior:

Use Retrofit to Upload an image to a Ktor server.
call receiveMultipart() to receive MP on the Ktor side.
See error

Expected behavior Multipart data should be received correctly.

Stack trace:

[nioEventLoopGroup-4-1] ERROR Application - Unhandled: POST - /api/upload
io.ktor.utils.io.charsets.MalformedInputException: Input length = 1
	at io.ktor.utils.io.charsets.CharsetJVMKt.throwExceptionWrapped(CharsetJVM.kt:323)
	at io.ktor.utils.io.charsets.CharsetJVMKt.decodeImplSlow(CharsetJVM.kt:289)
	at io.ktor.utils.io.charsets.CharsetJVMKt.decodeExactBytes(CharsetJVM.kt:254)
	at io.ktor.utils.io.core.StringsKt.readTextExactBytes(Strings.kt:293)
	at io.ktor.utils.io.core.StringsKt.readTextExactBytes$default(Strings.kt:292)
	at io.ktor.utils.io.core.AbstractInput.readText(AbstractInput.kt:468)
	at io.ktor.utils.io.core.AbstractInput.readText$default(AbstractInput.kt:465)
	at io.ktor.http.cio.CIOMultipartDataBase.partToData(CIOMultipartData.kt:127)
	at io.ktor.http.cio.CIOMultipartDataBase$partToData$1.invokeSuspend(CIOMultipartData.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:844)

ktor client does not log request body

https://youtrack.jetbrains.com/issue/KTOR-442

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1729

Ktor Version and Engine Used (client or server and name) ver: 1.3.1, 1.3.2 engine: Apache app: client dependencies:

dependencies {
    implementation(kotlin("stdlib-jdk8"))
    implementation(platform("io.ktor:ktor-bom:1.3.2"))
    implementation("io.ktor:ktor-client-core")
    implementation("io.ktor:ktor-client-logging-jvm")
    implementation("io.ktor:ktor-client-apache")
    implementation("io.ktor:ktor-client-json-jvm")
    implementation("org.slf4j:slf4j-simple:1.7.26")
    implementation("org.apache.httpcomponents:httpasyncclient:4.1.4")
}

Describe the bug Request body is not logged even with LogLevel.ALL

To Reproduce

package com.example

import io.ktor.client.HttpClient
import io.ktor.client.engine.apache.Apache
import io.ktor.client.features.logging.DEFAULT
import io.ktor.client.features.logging.LogLevel
import io.ktor.client.features.logging.Logger
import io.ktor.client.features.logging.Logging
import io.ktor.client.request.post
import io.ktor.util.KtorExperimentalAPI
import kotlinx.coroutines.runBlocking

@KtorExperimentalAPI
fun main() {
    runBlocking {
        val client = HttpClient(Apache) {
            install(Logging) {
                logger = Logger.DEFAULT
                level = LogLevel.ALL
            }
        }
        client.post<String>("http://localhost:8080") {
            body = """{ "test": "testVal" }"""
        }
    }
}

Expected behavior Request body is in the log

Actual result

[main] INFO io.ktor.client.HttpClient - REQUEST: http://localhost:8080/
[main] INFO io.ktor.client.HttpClient - METHOD: HttpMethod(value=POST)
[main] INFO io.ktor.client.HttpClient - COMMON HEADERS
[main] INFO io.ktor.client.HttpClient - -> Accept-Charset: UTF-8
[main] INFO io.ktor.client.HttpClient - -> Accept: */*
[main] INFO io.ktor.client.HttpClient - CONTENT HEADERS
[main] INFO io.ktor.client.HttpClient - BODY Content-Type: text/plain; charset=UTF-8
[main] INFO io.ktor.client.HttpClient - RESPONSE: 200 
---   remaining response stuff skipped for brevity  ---

Empty body in response using macosx64 target

https://youtrack.jetbrains.com/issue/KTOR-479

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1622

Ktor Version and Engine Used (client or server and name) 1.3.0, client (macosx64) (using curl)

Describe the bug When using macosx64 target in a Multiplatform project I'm getting empty BODY in response

HttpClient: REQUEST: http://api.open-notify.org/astros.json
HttpClient: METHOD: HttpMethod(value=GET)
HttpClient: COMMON HEADERS
HttpClient: -> Accept: application/json
HttpClient: -> Accept-Charset: UTF-8
HttpClient: CONTENT HEADERS
HttpClient: BODY Content-Type: null
HttpClient: BODY START
HttpClient: BODY END

The same code for iOS logs

HttpClient: REQUEST: http://api.open-notify.org/astros.json
HttpClient: METHOD: HttpMethod(value=GET)
HttpClient: COMMON HEADERS
HttpClient: -> Accept: application/json
HttpClient: -> Accept-Charset: UTF-8
HttpClient: CONTENT HEADERS
HttpClient: BODY Content-Type: null
HttpClient: BODY START
HttpClient: BODY END
HttpClient: BODY Content-Type: application/json
HttpClient: BODY START
HttpClient: {"people": [{"craft": "ISS", "name": "Andrew Morgan"}, {"craft": "ISS", "name": "Oleg Skripochka"}, {"craft": "ISS", "name": "Jessica Meir"}], "message": "success", "number": 3}
HttpClient: BODY END

The following is the common code that uses ktor

class PeopleInSpaceApi {
    private val url = "http://api.open-notify.org/astros.json"

    private val client by lazy {
        HttpClient() {
            install(JsonFeature) {
                serializer = KotlinxSerializer(Json(JsonConfiguration(strictMode = false)))
            }
            install(Logging) {
                logger = Logger.DEFAULT
                level = LogLevel.ALL
            }
        }
    }

    suspend fun fetchPeople(): AstroResult {
        return client.get(url) 
    }
}

I'm using following dependencies

        macOSMain.dependencies {
            // Coroutines
            implementation('org.jetbrains.kotlinx:kotlinx-coroutines-core-macosx64') {
                version {
                    strictly '1.3.3-native-mt'
                }
            }

            // Ktor
            implementation "io.ktor:ktor-client-curl:${Versions.ktor}"
            implementation "io.ktor:ktor-client-core-macosx64:${Versions.ktor}"
            implementation "io.ktor:ktor-client-json-macosx64:${Versions.ktor}"
            implementation "io.ktor:ktor-client-logging-macosx64:${Versions.ktor}"
            implementation "io.ktor:ktor-client-serialization-macosx64:${Versions.ktor}"

            // Serialize
            implementation "org.jetbrains.kotlinx:kotlinx-serialization-runtime-macosx64:${Versions.kotlinxSerialization}"

        }

(kotlinxSerialization = "0.14.0")

kotlinMultiplatform publications miss the gradle metadata

HttpClient can only be used on the main thread for native targets

https://youtrack.jetbrains.com/issue/KTOR-491

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1559

Ktor Version and Engine Used (client or server and name) Ktor client 1.2.6, default platform engine (iOS)

Describe the bug HttpClient can only be used from the main thread. It does not matter on which thread the object is instantiated.

The issue is caused by global declarations that are not annotated properly for Kotlin/Native.

Uncaught Kotlin exception: io.ktor.http.URLParserException: Fail to parse url: Caused by: kotlin.native.IncorrectDereferenceException: Trying to access top level value not marked as @ThreadLocal or @SharedImmutable from non-main thread at 0 test.kexe 0x0000000101edb8e7 kfun:kotlin.Throwable.(kotlin.String?)kotlin.Throwable + 87 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Throwable.kt:22:37) at 1 test.kexe 0x0000000101ed4ee5 kfun:kotlin.Exception.(kotlin.String?)kotlin.Exception + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:23:44) at 2 test.kexe 0x0000000101ed4aa5 kfun:kotlin.RuntimeException.(kotlin.String?)kotlin.RuntimeException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/Exceptions.kt:34:44) at 3 test.kexe 0x0000000101f07b55 kfun:kotlin.native.IncorrectDereferenceException.(kotlin.String)kotlin.native.IncorrectDereferenceException + 85 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/Runtime.kt:30:36) at 4 test.kexe 0x0000000101f25e99 ThrowIncorrectDereferenceException + 137 (/Users/teamcity3/buildAgent/work/4d622a065c544371/runtime/src/main/kotlin/kotlin/native/internal/RuntimeUtils.kt:87:11) at 5 test.kexe 0x0000000102200ed9 CheckIsMainThread + 25 at 6 test.kexe 0x00000001020e16a3 kfun:io.ktor.http.<get-URL_ALPHABET_CHARS>#internal + 35 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:11:9) at 7 test.kexe 0x00000001020e2ecd kfun:io.ktor.http.encodeURLPath@kotlin.String.()kotlin.String + 717 (/opt/buildAgent/work/a85294440dc5c6e/ktor-http/common/src/io/ktor/http/Codecs.kt:59:0) ...

I am using the CoroutineWorker library from Autodesk.

To Reproduce Steps to reproduce the behavior:

Make a client request from a non-main thread for a native target.

kotlin.native.concurrent.InvalidMutabilityException with 1.3.3-native-mt

https://youtrack.jetbrains.com/issue/KTOR-497

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1543

KTOR native with the new multi threaded coroutines fails with InvalidMutabilityException. coroutines version: 1.3.3-native-mt ktor version: 1.3.0-rc2 Here is the related code:

suspend fun doFetch() {
    HttpClient().use {
        val htmlContent = it.get<String>("https://en.wikipedia.org/wiki/Main_Page")
        println(htmlContent)
    }
}

fun doesNotWork() {
    runBlocking(Dispatchers.Default) { // remove Dispatchers.Default and it works
        doFetch()
    }
}

fun main() {
    doesNotWork()
}

to reproduce, checkout the following project: https://github.com/yigit/ktor-coroutines-bug

run ./gradlew runDELX --stacktrace from command line.

It fails with :

Uncaught Kotlin exception: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.utils.io.ByteChannelNative@c006ba18
        at kfun:kotlin.Throwable.<init>(kotlin.String?)kotlin.Throwable (0x2d9607)
        at kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception (0x2d3115)
        at kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException (0x2d2cd5)
        at kfun:kotlin.native.concurrent.InvalidMutabilityException.<init>(kotlin.String)kotlin.native.concurrent.InvalidMutabilityException (0x2fff65)
        at ThrowInvalidMutabilityException (0x301673)
        at MutationCheck (0x5a742e)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.<set-waitingForRead>#internal (0x403fed)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.$awaitSuspendCOROUTINE$73.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? (0x417d9f)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.awaitSuspend(kotlin.Int)ValueType (0x418244)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.$readRemainingSuspendCOROUTINE$60.invokeSuspend#internal (0x40f22c)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.readRemainingSuspend#internal (0x40f5c0)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.$readRemainingCOROUTINE$59.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? (0x40e936)
        at kfun:io.ktor.utils.io.ByteChannelSequentialBase.readRemaining(kotlin.Long;kotlin.Int)io.ktor.utils.io.core.ByteReadPacket (0x40ec6e)
        at kfun:io.ktor.utils.io.readRemaining@io.ktor.utils.io.ByteReadChannel.()io.ktor.utils.io.core.ByteReadPacket (0x421354)
        at kfun:io.ktor.client.features.HttpPlainText.Feature.$install$lambda-1COROUTINE$26.invokeSuspend#internal (0x53c9c4)
        at kfun:io.ktor.client.features.HttpPlainText.Feature.$install$lambda-1COROUTINE$26.invoke#internal (0x53d3ef)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x4d3e3d)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal (0x4d34be)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceedWith#internal (0x4d3697)
        at kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-1COROUTINE$22.invokeSuspend#internal (0x538612)
        at kfun:io.ktor.client.features.HttpCallValidator.Companion.$install$lambda-1COROUTINE$22.invoke#internal (0x538baf)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x4d3e3d)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal (0x4d34be)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.execute#internal (0x4d38c1)
        at kfun:io.ktor.util.pipeline.Pipeline.execute(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)#GENERIC_kotlin.Any (0x4ce690)
        at kfun:io.ktor.client.call.HttpClientCall.$receiveCOROUTINE$12.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? (0x5205d3)
        at kfun:io.ktor.client.call.HttpClientCall.receive(io.ktor.client.call.TypeInfo)kotlin.Any (0x520c06)
        at kfun:$doFetchCOROUTINE$0.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? (0x565423)
        at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) (0x2f4968)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x4d4508)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x4d4072)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x4d5d9c)
        at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) (0x2f4c49)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x4d4508)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x4d4072)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x4d5d9c)
        at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) (0x2f4c49)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith#internal (0x4d4508)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal (0x4d4072)
        at kfun:io.ktor.util.pipeline.SuspendFunctionGun.object-1.resumeWith#internal (0x4d5d9c)
        at kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) (0x2f4c49)
        at kfun:kotlinx.coroutines.DispatchedTask.run() (0x3e2fca)
        at kfun:kotlinx.coroutines.EventLoopImplBase.processNextEvent()ValueType (0x3bb507)
        at kfun:kotlinx.coroutines.runEventLoop$kotlinx-coroutines-core(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>) (0x3f665c)
        at kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.start$lambda-0#internal (0x3fe271)
        at kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.$start$lambda-0$FUNCTION_REFERENCE$80.invoke#internal (0x3fe3db)
        at kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.$start$lambda-0$FUNCTION_REFERENCE$80.$<bridge-UNN>invoke()#internal (0x3fe43b)
        at WorkerLaunchpad (0x30129c)
        at _ZN6Worker19processQueueElementEb (0x5ac1c9)
        at _ZN12_GLOBAL__N_113workerRoutineEPv (0x5aca2d)
        at  (0x7f54c8681c73)
        at clone (0x7f54c83b3def)
        at  ((nil))

Error Ktor running on background thread on iOS

https://youtrack.jetbrains.com/issue/KTOR-499

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1538

Ktor Version and Engine Used (client or server and name) Ktor client version 1.3.0-rc2 build.gradle:

ext.ktor_version = '1.3.0-rc2'
ext.kotlin_coroutines_version = '1.3.3-native-mt'

sourceSets {
        commonMain {
            dependencies {
                implementation kotlin('stdlib-common')
                implementation ("io.ktor:ktor-client-core:$ktor_version") {
                    exclude group: "org.jetbrains.kotlinx", module: "kotlinx-coroutines-core-common"
                }
                implementation "org.jetbrains.kotlinx:kotlinx-coroutines-core-common:$kotlin_coroutines_version"
            }
        }
        iosMain {
            dependencies {
                implementation ("io.ktor:ktor-client-ios:$ktor_version") {
                    exclude group: "org.jetbrains.kotlinx", module: "kotlinx-coroutines-core-native"
                    exclude group: "org.jetbrains.kotlinx", module: "kotlinx-coroutines-core-common"
                }
                implementation ("io.ktor:ktor-client-core-native:$ktor_version") {
                    exclude group: "org.jetbrains.kotlinx", module: "kotlinx-coroutines-core-native"
                    exclude group: "org.jetbrains.kotlinx", module: "kotlinx-coroutines-core-common"
                }
                implementation "org.jetbrains.kotlinx:kotlinx-coroutines-core-native:$kotlin_coroutines_version"
            }
        }
    }

Describe the bug Executing an API request on the background thread results in kotlin.native.concurrent.InvalidMutabilityException Full stacktrace

kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.client.request.HttpRequestPipeline@b5b748
        at 0   app                                 0x0000000105688a87 kfun:kotlin.Throwable.<init>(kotlin.String?)kotlin.Throwable + 87
        at 1   app                                 0x0000000105682215 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 85
        at 2   app                                 0x0000000105681dd5 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 85
        at 3   app                                 0x00000001056b6005 kfun:kotlin.native.concurrent.InvalidMutabilityException.<init>(kotlin.String)kotlin.native.concurrent.InvalidMutabilityException + 85
        at 4   app                                 0x00000001056b77d8 ThrowInvalidMutabilityException + 680
        at 5   app                                 0x00000001059a8c58 MutationCheck + 104
        at 6   app                                 0x00000001058c9400 kfun:io.ktor.util.pipeline.Pipeline.<set-interceptors>#internal + 96
        at 7   app                                 0x00000001058c98f5 kfun:io.ktor.util.pipeline.Pipeline.notSharedInterceptorsList#internal + 85
        at 8   app                                 0x00000001058c8a8a kfun:io.ktor.util.pipeline.Pipeline.cacheInterceptors#internal + 650
        at 9   app                                 0x00000001058c97a4 kfun:io.ktor.util.pipeline.Pipeline.sharedInterceptorsList#internal + 340
        at 10  app                                 0x00000001058c5f57 kfun:io.ktor.util.pipeline.Pipeline.createContext$ktor-utils(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)io.ktor.util.pipeline.PipelineExecutor<#GENERIC_kotlin.Any> + 263
        at 11  app                                 0x00000001058c5d71 kfun:io.ktor.util.pipeline.Pipeline.execute(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)#GENERIC_kotlin.Any + 273
        at 12  app                                 0x000000010590797a kfun:io.ktor.client.HttpClient.$executeCOROUTINE$6.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 666
        at 13  app                                 0x0000000105907cc4 kfun:io.ktor.client.HttpClient.execute(io.ktor.client.request.HttpRequestBuilder)io.ktor.client.call.HttpClientCall + 308
        at 14  app                                 0x000000010593bb7c kfun:io.ktor.client.statement.HttpStatement.$executeUnsafeCOROUTINE$35.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 956
        at 15  app                                 0x000000010593be8b kfun:io.ktor.client.statement.HttpStatement.executeUnsafe$ktor-client-core()io.ktor.client.statement.HttpResponse + 235
        at 16  app                                 0x000000010594b557 kfun:sample.ClientApi.$getQuote$lambda-0COROUTINE$0.invokeSuspend#internal + 6775
        at 17  app                                 0x00000001056a9ec8 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) + 712
        at 18  app                                 0x00000001057ca07c kfun:kotlinx.coroutines.DispatchedTask.run() + 2780
        at 19  app                                 0x000000010577fa18 kfun:kotlinx.coroutines.EventLoopImplBase.processNextEvent()ValueType + 792
        at 20  app                                 0x00000001057e3e31 kfun:kotlinx.coroutines.runEventLoop$kotlinx-coroutines-core(kotlinx.coroutines.EventLoop?;kotlin.Function0<kotlin.Boolean>) + 881
        at 21  app                                 0x00000001057eb412 kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.start$lambda-0#internal + 402
        at 22  app                                 0x00000001057eb5fb kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.$start$lambda-0$FUNCTION_REFERENCE$71.invoke#internal + 59
        at 23  app                                 0x00000001057eb65b kfun:kotlinx.coroutines.WorkerCoroutineDispatcherImpl.$start$lambda-0$FUNCTION_REFERENCE$71.$<bridge-UNN>invoke()#internal + 59
        at 24  app                                 0x00000001056b73a1 WorkerLaunchpad + 177
        at 25  app                                 0x00000001059b1909 _ZN6Worker19processQueueElementEb + 2569
        at 26  app                                 0x00000001059b1eb6 _ZN12_GLOBAL__N_113workerRoutineEPv + 54
        at 27  libsystem_pthread.dylib             0x00007fff51bfe2eb _pthread_body + 126
        at 28  libsystem_pthread.dylib             0x00007fff51c01249 _pthread_start + 66
        at 29  libsystem_pthread.dylib             0x00007fff51bfd40d thread_start + 13

To Reproduce Steps to reproduce the behavior:

Clone sample app from https://github.com/aleksey-chugaev/ktor-mt-app
Run iosApp in emulator
See error

Expected behavior Should successfully execute the request.

MPP JS client: Can't resolve 'text-encoding'

kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.client.request.HttpRequestPipeline@545e18f8

https://youtrack.jetbrains.com/issue/KTOR-693

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/887

Ktor Version

implementation("io.ktor:ktor-client-ios:1.1.1")

Ktor Engine Used(client or server and name)

Client

JVM Version, Operating System and Relevant Context

MBP 2017 JVM 1.8

Feedback

This exception was printed out,and the test can't print out the expected result

    suspend fun getDemoJSONDataFromNet(): String {


        val result = client.get<String>("https://en.wikipedia.org/wiki/Main_Page")

        Logger.d(tag, "the return data is $result")

        return result
    }

fun getDataIos(callBack: (result: String) -> Unit) {
    runBlocking {
       val job = async {
            val result = HttpUtil.getDemoJSONDataFromNet()
            Logger.d("SimpleIos","result value is $result")
            callBack(result)
        }
        job.await()
    }
}

    @Test
    fun testGetData() {
        getDataIos {
            println(it)
        }
    }

Exception Log:

Instances of kotlin.Error, kotlin.RuntimeException and subclasses aren't propagated from Kotlin to Objective-C/Swift.
Other exceptions can be propagated as NSError if method has or inherits @Throws annotation.
Uncaught Kotlin exception: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.client.request.HttpRequestPipeline@2ef2a08
        at 0   app                                 0x000000010dba2896 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 70
        at 1   app                                 0x000000010dba27b6 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 70
        at 2   app                                 0x000000010dba40e6 kfun:kotlin.native.concurrent.InvalidMutabilityException.<init>(kotlin.String)kotlin.native.concurrent.InvalidMutabilityException + 70
        at 3   app                                 0x000000010dc3c6b8 ThrowInvalidMutabilityException + 280
        at 4   app                                 0x000000010dc5ee48 MutationCheck + 24
        at 5   app                                 0x000000010dce8a83 kfun:io.ktor.util.pipeline.Pipeline.<set-interceptors>#internal + 67
        at 6   app                                 0x000000010dce86a6 kfun:io.ktor.util.pipeline.Pipeline.notSharedInterceptorsList#internal + 70
        at 7   app                                 0x000000010dce84d6 kfun:io.ktor.util.pipeline.Pipeline.cacheInterceptors#internal + 710
        at 8   app                                 0x000000010dce6292 kfun:io.ktor.util.pipeline.Pipeline.sharedInterceptorsList#internal + 98
        at 9   app                                 0x000000010dce61ca kfun:io.ktor.util.pipeline.Pipeline.createContext$ktor-utils(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)io.ktor.util.pipeline.PipelineExecutor<#GENERIC_kotlin.Any> + 90
        at 10  app                                 0x000000010dce60f0 kfun:io.ktor.util.pipeline.Pipeline.execute(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)#GENERIC_kotlin.Any + 96
        at 11  app                                 0x000000010dd313a8 kfun:io.ktor.client.HttpClient.$execute$COROUTINE$1.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 296
        at 12  app                                 0x000000010dd31522 kfun:io.ktor.client.HttpClient.execute(io.ktor.client.request.HttpRequestBuilder)io.ktor.client.call.HttpClientCall + 130
        at 13  app                                 0x000000010dd32696 kfun:io.ktor.client.call.$call$COROUTINE$3.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 758
        at 14  app                                 0x000000010dd32852 kfun:io.ktor.client.call.call@io.ktor.client.HttpClient.(kotlin.coroutines.SuspendFunction1<io.ktor.client.request.HttpRequestBuilder,kotlin.Unit>)io.ktor.client.call.HttpClientCall + 130
        at 15  app                                 0x000000010dd330eb kfun:io.ktor.client.call.call@io.ktor.client.HttpClient.(io.ktor.client.request.HttpRequestBuilder)io.ktor.client.call.HttpClientCall + 123
        at 16  app                                 0x000000010db97f3f kfun:sample.HttpUtil.$getDemoJSONDataFromNet$COROUTINE$0.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 1551
        at 17  app                                 0x000000010db97821 kfun:sample.HttpUtil.getDemoJSONDataFromNet()kotlin.String + 113
        at 18  app                                 0x000000010dba0de9 kfun:sample.$getDataIos$lambda-1$lambda-0$COROUTINE$3.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 313
        at 19  app                                 0x000000010dbd4801 kfun:kotlin.coroutines.native.internal.BaseContinuationImpl.resumeWith(kotlin.Result<kotlin.Any?>) + 385
        at 20  app                                 0x000000010dca7b89 kfun:kotlinx.coroutines.DispatchedTask.run() + 1129
        at 21  app                                 0x000000010dca92d6 kfun:kotlinx.coroutines.EventLoopImpl.processNextEvent()ValueType + 422
        at 22  app                                 0x000000010dccf7f7 kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 263
        at 23  app                                 0x000000010dccf5d7 kfun:kotlinx.coroutines.runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,#GENERIC>)Generic + 983
        at 24  app                                 0x000000010dccfb4c kfun:kotlinx.coroutines.runBlocking$default(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,#GENERIC>;kotlin.Int)Generic + 204
        at 25  app                                 0x000000010dba06e4 kfun:sample.getDataIos(kotlin.Function1<kotlin.String,kotlin.Unit>) + 100
        at 26  app                                 0x000000010dba0618 kfun:sample.main(kotlin.Array<kotlin.String>) + 216
        at 27  iosApp                              0x000000010d897774 $S6iosApp14ViewControllerC11viewDidLoadyyFTo + 36
        at 28  UIKitCore                           0x0000000112ea84e1 -[UIViewController loadViewIfRequired] + 1186

Release 1.4.1

Fix InvalidMutabilityException for tests in master

https://youtrack.jetbrains.com/issue/KTOR-1067

Native initialisation order leads to fail in constructor when we use this in field initialization in shared due to freeze:

kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen io.ktor.util.collections.internal.SharedForwardList@9fa908
    at kfun:kotlin.Throwable#<init>(kotlin.String?){} (0x2dc9b7)
    at kfun:kotlin.Exception#<init>(kotlin.String?){} (0x2d6b65)
    at kfun:kotlin.RuntimeException#<init>(kotlin.String?){} (0x2d67c5)
    at kfun:kotlin.native.concurrent.InvalidMutabilityException#<init>(kotlin.String){} (0x303585)
    at ThrowInvalidMutabilityException (0x304c53)
    at MutationCheck (0x5a519e)
    at kfun:io.ktor.util.collections.internal.SharedForwardList#<init>(){} (0x4ff407)
    at kfun:io.ktor.util.collections.ConcurrentMap#<init>(io.ktor.util.Lock;kotlin.Int){} (0x4e9814)
    at kfun:io.ktor.util.collections.ConcurrentMap#<init>(io.ktor.util.Lock?;kotlin.Int;kotlin.Int;kotlin.native.internal.DefaultConstructorMarker?){} (0x4e9af2)
    at kfun:io.ktor.util.ConcurrentMapTest#testEmpty(){} (0x52e1a5)
    at kfun:io.ktor.util.$ConcurrentMapTest$test$0.$testEmpty$FUNCTION_REFERENCE$14.invoke#internal (0x52f893)
    at kfun:io.ktor.util.$ConcurrentMapTest$test$0.$testEmpty$FUNCTION_REFERENCE$14.$<bridge-UNNN>invoke(-1:0){}#internal (0x52f909)
    at kfun:kotlin.native.internal.test.BaseClassSuite.TestCase#run(){} (0x32755b)
    at kfun:kotlin.native.internal.test.TestRunner.run#internal (0x3215b8)
    at kfun:kotlin.native.internal.test.TestRunner.runIteration#internal (0x32276e)
    at kfun:kotlin.native.internal.test.TestRunner#run(){}kotlin.Int (0x3230a5)
    at kfun:kotlin.native.internal.test#testLauncherEntryPoint(kotlin.Array<kotlin.String>){}kotlin.Int (0x316eec)
    at kfun:kotlin.native.internal.test#main(kotlin.Array<kotlin.String>){} (0x316db7)
    at Konan_start (0x31701b)

There is also usages of simple references instead of shared in tests and client capabilities.

Changelog 1.4 version

1.4.3

Client

iOS client fails with CoroutinesInternalError when Logging is used

CIO: client engine exceptions are both logged and thrown

Timeout﻿ feature: android engine throws Java's SocketTimeoutException instead of ConnectTimeoutException

HttpRedirect feature alters Location header value

CIO Engine's HttpClient may fail when trying to send large size binary data.

Client: NPE in FormDataContentKt -> Input.copyTo

Core

Experimental API and compatibility guarantees

Input.readTextExactBytes(n) on empty input different behavior per platform

ktor-io: JVM shared function decrease performance starting from 1.4.0

Infrastructure

Add build parameter to build ktor with JVM IR compiler

CI does not trigger build on PR to samples repo

Invalid publication 'kotlinMultiplatform' for task :publishKotlinMultiplatformPublicationToMavenLocal on master

Server

HTTP/2 push fails with netty engine

Request headers exceeding expected threshold are not handled correctly

Cannot receive content via jackson negotiator since 1.4.2

Sessions + SSL (Netty)

Netty HTTP/2 HEAD response hangs

Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.

"Wrong HEX escape": gracefully handle invalid URLs

Upgrade Netty to 4.1.54.Final

Other

Client: URL encode / escaping is wrong

Wrong pool is used to release `IOBuffer` after `ByteChannelSequential.copyTo` from static initialized instance.

`ByteBufferChannel.readRemaining` doesn't read whole channel

Update kotlin to 1.4.20

Fix configuration if project without VPN and cache

Handle failure in reading request body

Remove copyTo usage from ServerPipeline

1.4.2

Client

Ktor Client HTTP Cache for invalid or blank Expires header

Content-type header is missing in Request when using LogLevel.BODY

NoTransformationFoundException when using JsonFeature and connecting through a proxy server

"IllegalStateException: Cannot execute task because event loop was shut down" for native client

Native: ResponseObserver hangs coroutine indefinitely

Client: InvalidCacheStateException when Vary header differs for 200 and 304 responses

IncorrectDereferenceException thrown when ktor request call from other thread

Jetty client engine with high concurrency randomly freezes, throws NPEs, ISEs or EofExceptions

Ktor 1.4.1 fails to upload file (submit form) when logging level = LogLevel.BODY

CertificatePinner buildErrorMessage outputs CertificatesInfo reference and variable name (HASH_ALGORITHM_SHA_256) instead of value

Overload resolution ambiguity: HttpClient.ws

Core

CIO client engine randomly runs into infinite loop under high load

Infrastructure

ktor-server-test-host indirectly depends on logback-classic

Add artifact dependency to release configuration

Plugin

Test compilation errors when project created via plugin with "Mock HttpClient Engine"

No changelog for 1.4.0 and 1.4.1 on the plugin page in IDE Marketplace

Plugin: the Ktor project type tooltip in the New Project wizard contains text used for Java projects

Plugin: "Could not find io.ktor:ktor-client-http-timeout" when building project created with http-timeout feature

Server

Class cast exception on web-socket cancelation

CORS - request with non-simple content type not failing

Scheme instead of version in OriginConnectionPoint

Setting multiple cache control directives is impossible with current API

Sensible defaults for compression feature

Netty: NumberFormatException when make a request via HTTP/2 without explicit port

Make Jackson serialization stream response

Test Infrastructure

Replace JUnit with kotlin.test

Other

TestHttpClientEngine KNPE

Ktor Client Null Pointer Exception (HTTPS)

Ktor CIO client, connect timeout handling

Url(…) doesn't parse non-HTTP-like URL strings correctly

"response.readText()" never returns (probably infinite loop)

Create sample with native CIO client and `native-mt` coroutines

Split ktor-server-test-host module

Extract flaky tests from `master` to the `flaky-tests` branch

Blocking calls (ReentrantLock.lock) on Apache engine dispatcher threads

Fix HTTP2 tests with new versions of java8

1.4.1

Client

Timeout feature: android engine throws Java's SocketTimeoutException instead of ConnectTimeoutException