Changelog 1.5 version

1.5.3

released 5th April 2021

Client

Outdated doc string for FormPart

Ktor 1.5.2: Can't resolve 'node-fetch' on libs produced by jsBrowserProductionLibraryDistribution

Fix flaky CIOSustainabilityTest.testBlockingConcurrency[jvm]

java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey

https://youtrack.jetbrains.com/issue/KTOR-2389

After updating from ktor 1.4.3 to 1.5.2 and running my integration tests using ktor I'm running into:

java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
	at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
	at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
	at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
	at ???(Coroutine boundary.?(?)
	at app.becoach.roboter.CoacheeRoboter$login$1.invokeSuspend(CoacheeRoboter.kt:26)
Caused by: java.lang.IllegalStateException: No instance for key AttributeKey: ExpectSuccessAttribyteKey
	at io.ktor.util.Attributes$DefaultImpls.get(Attributes.kt:29)
	at io.ktor.util.AttributesJvmBase.get(AttributesJvm.kt:15)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:28)
	at io.ktor.client.features.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
	at io.ktor.client.features.HttpCallValidator.validateResponse(HttpCallValidator.kt:55)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:134)
	at io.ktor.client.features.HttpCallValidator$Companion$install$3.invoke(HttpCallValidator.kt)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:98)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Do you happen to have any idea what's causing this? My unit tests are getting 4xx error codes from the backend. I've got a custom Feature installed where upon unauthorized error codes, I'm re-issuing a new token.

My feature looks like this:

internal class AuthFeature(
  private val credentials: String,
  private val tokenStorage: TokenStorage,
  private val logger: Logger
) {
  class Config {
    lateinit var credentials: String
    lateinit var tokenStorage: TokenStorage
    lateinit var logger: Logger
  }

  companion object Feature : HttpClientFeature<Config, AuthFeature> {
    override val key: AttributeKey<AuthFeature> = AttributeKey("AuthFeature")

    /**
     * The Mutex is required in order to only refresh once when multiple requests
     * are being triggered simultaneously e.g. at app startup.
     */
    private val mutex = Mutex()

    override fun prepare(block: Config.() -> Unit): AuthFeature {
      val config = Config().apply(block)
      return AuthFeature(
          credentials = config.credentials,
          tokenStorage = config.tokenStorage,
          logger = config.logger,
      )
    }

    @KtorExperimentalAPI
    override fun install(feature: AuthFeature, scope: HttpClient) {
      scope.requestPipeline.intercept(HttpRequestPipeline.State) {
        val accessToken = feature.tokenStorage.getAccessToken()
        val isRefreshingToken = context.url.buildString().endsWith("oauth/token")
        val header = if (accessToken != null && !isRefreshingToken) "Bearer $accessToken" else feature.credentials
        context.headers.append(HttpHeaders.Authorization, header)
      }

      requireNotNull(scope.feature(HttpSend)).intercept { call, _ ->
        if (call.response.status == HttpStatusCode.Unauthorized) {
          val mutexId = uuid4().toString()
          val tag = "${key.name}-$mutexId"
          feature.logger.i(tag, "Got ${HttpStatusCode.Unauthorized.value}")

          val beforeMutexAccessToken = feature.tokenStorage.getAccessToken()
          val request = call.request

          mutex.withLock {
            val inMutexAccessToken = feature.tokenStorage.getAccessToken()
            val needsToRefresh = beforeMutexAccessToken == inMutexAccessToken

            if (needsToRefresh) {
              feature.tokenStorage.putAccessToken(null) // Invalidate access token.
            }

            val refreshToken = feature.tokenStorage.getRefreshToken()

            if (refreshToken != null) {
              if (needsToRefresh) {
                feature.logger.i(tag, "Issuing new access token")
                val refreshResponse = scope.refreshToken(refreshToken)
                feature.logger.i(tag, "Updating access & refresh token")

                feature.tokenStorage.putAccessToken(refreshResponse.accessToken)
                feature.tokenStorage.putRefreshToken(refreshResponse.refreshToken)
              } else {
                feature.logger.i(tag, "Other request already refreshed token")
              }

              // Build previous request once again.
              val requestBuilder = HttpRequestBuilder()
              requestBuilder.takeFrom(request)

              // Force the new token by explicitly setting it.
              requestBuilder.headers[HttpHeaders.Authorization] = "Bearer ${feature.tokenStorage.getAccessToken()}"

              feature.logger.i(tag, "Retrying previous request with new access token")
              execute(requestBuilder).also {
                feature.logger.i(tag, "Finished previous request")
              }
            } else {
              feature.logger.i(tag, "No refresh token and hence leaving as is")
              call
            }
          }
        } else {
          call
        }
      }
    }
  }
}

Core

Upgrade kotlinx.serialization to 1.1.0

Upgrade to coroutines 1.4.3

I/O readRemaining sometimes looses exception

Docs

Wrong redirect from old documentation page

404 page on https://ktor.io/clients/http-client.html#websockets

Wrong indentation in `Serving Static Content` guide

Doc Code Snippets contain out of data repositories

Validate tutorial on Creating HTTP APIs with Ktor

Generator

A project generated with Maven using a new plugin contains deprecated API in 'pom.xml'

Provide default route in generated application

Wizard generates obsolete repositories

Rename build system options in generator

Infrastructure

Autoreload not working with 1.5.x when using embeddedServer NOT in debug mode

Plugin

Application structure browsing, routing navigator and edit for Ktor plugin

New Ktor plugin saves scroll position in feature description

Documentation links don't work in a plugin

Reference-style documentation links are not rendered in a plugin

Remove README From generated projects

Link to GitHub for Features is incorrect

Wizard: "Static Content" feature generates invalid index.html

Wizard: WebSocket feature adds redundant qualifier name

Wizard: Creating Maven project with Kotlinx.Serialization results in Could not find artifact org.jetbrains.kotlin:kotlin-maven-serialization:pom

Wizard: Gradle Groovy project with Kotlinx Serialization uses inconsistent quotes

Wizard: Basic run configuration is not always created

Wizard: Unused import directive in build.gradle.kts

Wizard: Selecting MAVEN adds multiple Bintray references to pom.xml

https://youtrack.jetbrains.com/issue/KTOR-2300

For some reason it also add numerous entries

<repositories>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>jcenter</id>
            <url>https://jcenter.bintray.com</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>ktor</id>
            <url>https://kotlin.bintray.com/ktor</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

Server

CORS

Simple API for writing features

https://youtrack.jetbrains.com/issue/KTOR-929

Main goals

Make API easy to understand for users: a) Get rid of concept of pipelines and phases b) Make API compact (reduce number of options available, i.e. consider smaller number of "phases") c) Rework naming to make it intuitive d) Think about defaults and yet support common case
API must be compatible with old existing features
All usages of old features rewritten with new API must remain unchanged yet working. This means all import {...} blocks should still work for features rewritten to the new API as well as with the old features.
Choosing "phase" must become intuitive and easy
Old API functionality should remain available for experienced users.
API should fit into Kotlin best patterns and concepts (DSL is a good option here)

Onboarding (first steps)

Old API

Let's take a look at some small feature that was defined with the old Features API. This example was taken from our official documentation:

class CustomFeature(configuration: Configuration) {
    val prop = configuration.prop // Copies a snapshot of the mutable config into an immutable property.

    class Configuration {
       var prop = "value" // Mutable property.
    }

    // Implements ApplicationFeature as a companion object.
    companion object Feature : ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature> {
       // Creates a unique key for the feature.
       override val key = AttributeKey<CustomFeature>("CustomFeature")

       // Code to execute when installing the feature.
       override fun install(pipeline: ApplicationCallPipeline, configure: Configuration.() -> Unit): CustomFeature {

           // It is responsibility of the install code to call the `configure` method with the mutable configuration.
           val configuration = CustomFeature.Configuration().apply(configure)

           // Create the feature, providing the mutable configuration so the feature reads it keeping an immutable copy of the properties.
           val feature = CustomFeature(configuration)

           // Intercept a pipeline.
           pipeline.intercept(…) {
                // Perform things in that interception point.
           }
           return feature
       }
    }
}

Old Problems:

This code does not look welcoming to newcomers at all.
It is not easy to understand.
It contains a lot of concepts to deal with that are not hidden by the API: ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature>, AttributeKey<CustomFeature>, ApplicationCallPipeline, pipeline.intercept
It is not clear where is the feature itself and where is it's configuration: there is a CustomFeature that has some properties, and there is also a Configuration class that defines configuration, and there is also a Feature object. It is not 100% obvious how each of them is used and such model confuses unexperienced readers because there is no clear separation between these concepts.

Proposed solution:

Instead of creating 2 nested classes and an object, extending some weird class ApplicationFeature<ApplicationCallPipeline, CustomFeature.Configuration, CustomFeature> now it is sufficient to just create an instance of KtorFeature using createFeature { ... } DSL method, and to define a configuration class for it separately:

class Configuration {
    // Properties and configuration functions go here:
    ...
}

val CustomFeature = createFeature(name = "CustomFeature", createConfiguration = ::Configuration) {
    // What feature does goes here:
    ...
}

[update]: cc @leonid.stashevsky createFeature block can be basically called anywhere as long as it makes sense for it to be visible by the scope of install that uses it's result. In other words, createFeature block creates an object (and it makes sense to just make it a global object) of type KtorFeature and this object can be used as an argument of install(...).

This feature can now be used as it was before, with just calling install and configuring it in your main application code:

install(CustomFeature) {
    // set mutable config properties
}

Benefits

Now, users will see much more clear separation between the data and actions, and they will not have to deal with confusing methods and base classes to extend.
It is fully compatible with the old code that uses features

Extending Ktor HTTP behaviour

Old API

In the old API users had to deal with a concept of phases and pipelines. Below there is a list of them:

ApplicationCallPipeline
- Setup : Phase for preparing call and it's attributes for processing
- Monitoring : Phase for preparing call and it's attributes for processing
- Features : Phase for features. Most features should intercept this phase.
- Call : Phase for processing a call and sending a response
- Fallback : Phase for handling unprocessed calls
ApplicationSendPipeline
- Before : The earliest phase that happens before any other
- Transform : Transformation phase that can proceed with any supported data like String
- Render : Phase to render any current pipeline subject into [io.ktor.http.content.OutgoingContent], beyond this phase only [io.ktor.http.content.OutgoingContent] should be produced by any interceptor
- ContentEncoding : Phase for processing Content-Encoding, like compression and partial content
- TransferEncoding : Phase for handling Transfer-Encoding, like if chunked encoding is being done manually and not by engine
- After : The latest application phase that happens right before engine will send the response
- Engine : Phase for Engine to send the response out to client.
ApplicationReceivePipeline
- Before : Executes before any transformations are made
- Transform : Executes transformations
- After : Executes after all transformations

Note, that in the list above phases are ordered!

Also, each feature had an ability to create it's custom phase in any of the 3 pipelines, and insert this phase anywhere in the order of existing phases. In order to use some phase ("intercept a pipeline in a specific phase") a feature was meant to call pipeline.intercept method on a specific phase.

Here is an example of usage of the old API taken from the MicrometerMetrics implementation:

 override fun install(pipeline: Application, configure: Configuration.() -> Unit): MicrometerMetrics {
    ...
    val phase = PipelinePhase("MicrometerMetrics")
    pipeline.insertPhaseBefore(ApplicationCallPipeline.Monitoring, phase)

    pipeline.intercept(phase) {
        // some actions go here
    }
    val postSendPhase = PipelinePhase("MicrometerMetricsPostSend")
    pipeline.sendPipeline.insertPhaseAfter(ApplicationSendPipeline.After, postSendPhase)
    pipeline.sendPipeline.intercept(ApplicationSendPipeline.After) {
        // some actions go here
    }

Old problems:

There too many phases and pipelines (3 pipelines with 15 phases total)
Not obvious which phase to select,
No defaults for some simple usage (onboarding)
With user-defined phases and insertBefore/insertAfter it builds a complex mental model of the global order of phases and requires a deep knowledge about phases each feature uses in order to think about the order of execution.
Concept of phases and pipelines is not intuitive and does not mentally map to the concept of Routing in Ktor. It is not obvious when each phase is executed and what are the pipelines.

Proposed solution:

First of all, there should be simple defaults that would extend basic functionalities that users see in Ktor. There will be 3 "default" methods that are mapped by name with what call inside a routing block each of them extend:

onCall : extends what happens when a server is processing any HTTP method call
onReceive : extends behaviour of the server when user writes call.receive() in a route { ... } block
onRespond : extends behaviour of the server when user writes call.respond() (call.respondText(), etc.) in a route { ... } block

In code they should be placed inside createFeature block:

val F = createFeature(...) {
    onCall {
        ...
    }
    onReceive {
        ...
    }
    onRespond {
        ...
    }
}

Such defaults are useful for newbies because they are meant to provide the most commonly used functionality without a requirement to know a lot about other options.

Also, for some more advanced use cases there will be more advanced builders with more options to extend:

extendCallHandling { ... } -- a block that configures what happens when HTTP method call handling
- monitoring { ... } -- defines actions to perform before the call was processed by any feature (including Routing). It is useful for monitoring and logging (see CallLogging feature) to be executed before any "real stuff" was performed with the call because other features can change it's content as well as add more resource usage etc. while for logging and monitoring it is important to observe the pure (untouched) data.
- onCall { ... } -- same as onCall outside of extendCallHandling. Most used extension. It defines what to do with the call that is being currently processed by a server.
- fallback -- defines what to do with the call in case request was not handled for some reason. Usually, it is handy to use fallback { call -> ...} to set the response with some message or status code, or to throw an exception.
extendReceiveHandling supports following callbacks:
- beforeReceive — defines actions to perform before any transformations were made to the received content. It is useful for caching.
- onReceive — defines how to transform data received from a client.
extendResponseHandling supports following callbacks:
- beforeResponse — allows to use the direct result of call processing (see onCall)
- onResponse — does transformations of the data. Example: you can write a custom serializer wit h it.
- afterResponse — allows to calculate some stats on the data that was sent, or to handle errors. It is used in Metrics, CachingHeaders, StatusPages.

Example:

val F = createFeature(...) {
    onCall {
        // prepare some data
    }
    extendResponseHandling {
        beforeResponse {
            // use this data
        }
        onResponse {
            // transform data to a string (i.e. serialize data)
        }
    }
}

Benefits:

Now we provide fast-to-access defaults that newbies can use (now it is obvious what to select from: not sure -- use a default)
There are much less options to select from, and these options are mapped to what users do when they define handling of routes
Api doesn't introduce any new complex or uncommon concepts (like phases or pipelines, or interceptors)

Managing order of execution between features

Old API

In the old API there was a concept of phases that one could introduce and then insert anywhere with insertPhaseBefore / insertPhaseAfter, and then intercept these phases. One of the most common things it was used for was creating order dependencies between features. This idea is explained here: https://www.ximedes.com/2020-09-17/role-based-authorization-in-ktor/. This was useful to either prepare some data that can be used later by another feature, or to use some data already produced by some other feature.

Old Problems:

It required using 3 concepts to do 1 thing: use "phase", "insertPhase" and "intercept" in order to "execute one feature before/after another"
It required explicitly knowing which phases another feature was using.

Proposed solution:

Each feature now internally remembers which phases in which pipelines it intercepts (of course, without revealing this complex and confusing information to users). And createFeature block also provides following functions inside:

beforeFeature(other) { ... }
afterFeature(other) { ... }

Example:

val F = createFeature(...) {
    ...
}

val G = createFeature(...) {
    beforeFeature(F) {
        onCall {
            // prepare some data for feature F
        }
    }
}

Benefits:

Now the API just does what it says: it is simply executing one feature before/after another without introducing any complex entities and without a requirement to manually manage the internals.

Manually managing features execution:

Old API

In the old API it was possible to manually manage execution of the current pipeline. Inside any interceptor it was possible to call the following functions of PipelineContext<, >:

public interface PipelineContext<TSubject : Any, TContext : Any> : CoroutineScope {
    /**
     * Object representing context in which pipeline executes
     */
    public val context: TContext

    /**
     * Subject of this pipeline execution that goes along the pipeline
     */
    public val subject: TSubject

    /**
     * Finishes current pipeline execution
     */
    public fun finish()

    /**
     * Continues execution of the pipeline with the given subject
     */
    public suspend fun proceedWith(subject: TSubject): TSubject

    /**
     * Continues execution of the pipeline with the same subject
     */
    public suspend fun proceed(): TSubject
}

Also, there were some extension methods over this class:

application : Current application for the context
call : Current call for the context
transformDefaultContent : Default outgoing content transformation

Although it introduced really complex concept of subject to understand, it can be required for some advanced users and we can't get rid of it completely. One of the most popular patterns that use PipelineContext is the following (taken from VelocityFeature):

pipeline.sendPipeline.intercept(ApplicationSendPipeline.Transform) { value ->
    if (value is VelocityContent) {
        val response = feature.process(value)
        proceedWith(response)
    }
}

It actually allows to write a custom data serializer for a specific type (VelocityContent in this example). proceedWith(outgoingContent) allows to continue the execution of our HTTP pipeline with the data serialized to a resulting outgoingContent.

Old problems

Really complex and hard to understand but yet useful
Some members/extensions are not that popular or not required for the API (application is not commonly used, context makes no sense to newbies)
proceedWith and suject are useful but are mostly used for serialization features
serialization use case requires knowing about OutgoingContent type that is not mapped to any general knowledge, and in fact there is always a default serializer from String, ByteChanel and ByteArray types to the OutgoingContent.

Proposed solution:

For the case of serialization we will introduce following simple API that users may default to:

public fun <T : Any> KtorFeature<T>.serializeDataToString(callback: suspend SendExecution.(Any) -> String?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

public fun <T : Any> KtorFeature<T>.serializeDataToBytes(callback: suspend SendExecution.(Any) -> ByteArray?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

public fun <T : Any> KtorFeature<T>.serializeDataToChanel(callback: suspend SendExecution.(Any) -> ByteWriteChannel?): Unit =
    onSend { callback(subject)?.let { proceedWith(it) } }

This API can be used as following:

private class Formula(val x: String, val execute: (String) -> String)

private val F = createFeature("F", {}) {
    serializeDataToString { f ->
        // return null for irrelevant data types
        if (f !is Formula) return@serializeDataToString null

        // Do a transformation for the current data type
        f.execute(f.x)
    }
}

private fun Application.f() {
    routing {
        get("/") {
            call.respond(Formula("cake") { x -> "x * x + 1".replace("x", x) })
        }
    }
}

Also, for any more complex use cases we will introduce the following wrapper over PipelineContext<SubjectT, ApplicationCall>. It is meant to hide unneeded methods and extensions without limiting functionality too much:

public inline class Execution<SubjectT : Any>(private val context: PipelineContext<SubjectT, ApplicationCall>) {
    public suspend fun proceed(): SubjectT = context.proceed()
    public suspend fun proceedWith(subectT: SubjectT): SubjectT = context.proceedWith(subectT)
    public fun finish(): Unit = context.finish()
    public val subject: SubjectT get() = context.subject
    public val call: ApplicationCall get() = context.call
...

[update] cc @leonid.stashevsky Also following methods can be added to handle application-level properties from a feature:


    // Useful methods
    public val environment: ApplicationEnvironment get() = context.application.environment
    public val configuration: ApplicationConfig get() = environment.config
    public val port: Int get() = configuration.propertyOrNull("ktor.deployment.port")?.getString()?.toInt() ?: 8080
    public val host: String get() = configuration.propertyOrNull("ktor.deployment.host")?.getString() ?: "0.0.0.0"
    // Following calback can be used to close resources that a feature has allocated (if needed):
     public fun onShutdown(callback: suspend () -> Unit) {
        GlobalScope.launch(context.coroutineContext) {
            callback()
        }
    }
}

Also there are type aliases for executions of the different stages:

public typealias CallExecution = Execution<Unit>
public typealias ReceiveExecution = Execution<ApplicationReceiveRequest>
public typealias SendExecution = Execution<Any>

Each of the methods has a corresponding Execution type as a reciever for a callback, for example:

public override fun onCall(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit { ... }

This means, that inside a feature definition advanced users who know what they do may use proceed, finish etc. methods anywhere in the code:

val F = createFeature(...) {
    extendRespondHandling {
        beforeRespond {
            ...
            if (someCondition)
                finish()
            ...
        }
    }
}

[update] cc @leonid.stashevsky Another option for an API

There are following minor issues with the proposed API:

Naming is quite long for advanced options (extendCallHandling as opposed to onCall), although it is made so intentionally to hide advanced settings from newbies by a longer name.
It sometimes requires writing thisin Intellij in order to call a completion to learn what's next.

Benefits

Now it is easy to write serializers
Now advanced users can still use what they want
No complex classes are in your face when you write simple code, so nothing will confuse newbies.

Alternative

We can introduce following interfaces (usage will be described later):

public interface OnCall {
    public operator fun invoke(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
    public fun monitoring(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
    public fun fallback(callback: suspend CallExecution.(ApplicationCall) -> Unit): Unit
}

public interface OnReceive {
    public operator fun invoke(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
    public fun before(callback: suspend ReceiveExecution.(ApplicationCall) -> Unit): Unit
}

public interface OnResponse {
    public operator fun invoke(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
    public fun before(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
    public fun after(callback: suspend ResponseExecution.(ApplicationCall) -> Unit): Unit
}

And change FeatureContext (a supertype of KtorFeature) to the following:

public interface FeatureContext {
    public val onCall: OnCall
    public val onReceive: OnReceive
    public val onResponse: OnResponse
}

After implementing FeatureContext it will allow us to write the code in the next way:

val F = makeFeature("F", {}) {

    onCall { ... }

    onCall.fallback { ... }
    onCall.monitoring { ...}

    beforeFeature(G) {
        onCall { ... }
        onReceive.before { ... }
        onResponse.after { ... }
    }
    ...

So, in general, following API transformations have been applied:

onCall { ... } -> onCall { ... }
extendCallExecution { fallback { ... } } -> onCall.fallback { ... }
extendCallExecution { monitoring { ... } } -> onCall.monitoring { ... }
onRespond { ... } -> onRespond { ... }
extendRespondExecution { before { ... } } -> onRespond.before { ... }
extendRespondExecution { after { ... } } -> onRespond.after { ... }
onReceive { ... } -> onReceive { ... }
extendReceiveExecution { before { ... } } -> onReceive.before { ... }

Benefits

Much shorter and readable names
Advanced functionality is still hidden (now it is hidden by the dot .)
Completion perfectly works when one types a dot after an object name to see the options
Simple things are kept the same because onCall, onReceive and onSend are executable objects now

Disadvantages

Not conventional way of writing API, may be uncommon to write dots after kinda-function names

[update] cc @leonid.stashevsky Compatibility with features written in old API

New API provides an ability for features to interact/depend on each other with beforeFeature and afterFeature.

Inside it automatically creates everything that is needed to guarantee the order between features. But this change requires old features to imlement new interface InterceptionsHolder in order to make it working with the old features. We can easily do that for builtin features:

/**
 * Compatibility class. Interception class for Call phase
 * */
public typealias CallInterception = Interception<Unit>
/**
 * Compatibility class. Interception class for Receive phase
 * */
public typealias ReceiveInterception = Interception<ApplicationReceiveRequest>
/**
 * Compatibility class. Interception class for Send phase
 * */
public typealias ResponseInterception = Interception<Any>

/**
 * Compatibility class. Every feature that needs to be compatible with `beforeFeature(...)` and `afterFeature(...)`
 * needs to implement this class. It defines a list of interceptions (see [Interception]) for phases in different pipelines
 * that are being intercepted by the current feature.
 * 
 * It is needed in order to get first/last phase in each pipeline for the current feature and create a new phase
 * that is strictly earlier/later in this pipeline than any interception of the current feature.
 * 
 * Note: any [KtorFeature] instance automatically fills these fields and there is no need to implement them by hand.
 * But if you want to use old feature API based on pipelines and phases and to make it available for `beforeFeature`/
 * `afterFeature` methods of the new features API, please consider filling the phases your feature defines by implementing
 * this interface [InterceptionsHolder].
 * */
public interface InterceptionsHolder {
    public val name: String get() = this.javaClass.simpleName

    public val fallbackInterceptions: MutableList<CallInterception>
    public val callInterceptions: MutableList<CallInterception>
    public val monitoringInterceptions: MutableList<CallInterception>

    public val beforeReceiveInterceptions: MutableList<ReceiveInterception>
    public val onReceiveInterceptions: MutableList<ReceiveInterception>

    public val beforeResponseInterceptions: MutableList<ResponseInterception>
    public val onResponseInterceptions: MutableList<ResponseInterception>
    public val afterResponseInterceptions: MutableList<ResponseInterception>

    public fun newPhase(): PipelinePhase = PipelinePhase("${name}Phase${Random.nextInt()}")
}

Question: should we make it internal and hide

pros:

It reveals old API abstactions and is really complex
It has a lot of fields to implement and is confusing
We can implement it in our builtin features and this will cover almost all the cases of the features interaction. It is very rare when someone wants to insert their feature before/after some other third-party feature while it is much more common to insert one relatively to some builtin feature. For the rest of the cases, we can suggest using new API that is easier and does all the work automatically.

cons:

(Didn't actually find any use-case) probably we'll force old users to rewrite their features with the new API to make them available to the new features.

Unexpected exception when using Session feature: "Using blocking primitives on this dispatcher is not allowed"

https://youtrack.jetbrains.com/issue/KTOR-1452

kotlin: 1.4.20 kotlinx.coroutines: 1.4.2

Sessions feature is configured in a rather standard manner:

        install(Sessions) {
            cookie<UserIdPrincipal>(
                Cookies.SESSION_ID,
                storage = directorySessionStorage(sessionStorageRootDir)
            ) {
                cookie.path = "/"
                cookie.httpOnly = true
                cookie.extensions["SameSite"] = "Lax"
            }
        }

When it is called during a pipeline execution, an unexpected exception is thrown.

java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
	at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	(Coroutine boundary) ~[?:?]
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:259) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) ~[ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) ~[ktor-server-netty-jvm-1.4.3.jar:1.4.3]
Caused by: java.lang.IllegalStateException: Using blocking primitives on this dispatcher is not allowed. Consider using async channel instead or use blocking primitives in withContext(Dispatchers.IO) instead.
	at io.ktor.utils.io.jvm.javaio.BlockingKt.ensureParkingAllowed(Blocking.kt:302) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.access$ensureParkingAllowed(Blocking.kt:1) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.InputAdapter.<init>(Blocking.kt:30) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.utils.io.jvm.javaio.BlockingKt.toInputStream$default(Blocking.kt:20) ~[ktor-io-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.DirectoryStorage.read(DirectoryStorage.kt:46) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invokeSuspend(CacheStorage.kt:15) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage$referenceCache$1.invoke(CacheStorage.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invokeSuspend(Cache.kt:82) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache$container$1.invoke(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2$1.invokeSuspend(Cache.kt:38) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.4.20.jar:1.4.20-release-308 (1.4.20)]
	at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith(DispatchedContinuation.kt:342) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable$default(Cancellable.kt:27) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:109) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async(Builders.common.kt:91) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.async(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.async$default(Builders.common.kt:84) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:37) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseCache$getOrCompute$2.apply(Cache.kt:31) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_251]
	at io.ktor.sessions.BaseCache.getOrCompute(Cache.kt:36) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache.getOrCompute$suspendImpl(Cache.kt:86) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.ReferenceCache.getOrCompute(Cache.kt) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.BaseTimeoutCache.getOrCompute(Cache.kt:163) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.CacheStorage.read(CacheStorage.kt:20) ~[ktor-server-sessions-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionTrackerById.load(SessionTrackerById.kt:34) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.SessionsKt.receiveSessionData(Sessions.kt:215) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invokeSuspend(Sessions.kt:63) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.sessions.Sessions$Feature$install$1.invoke(Sessions.kt) ~[ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:110) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invokeSuspend(StatusPages.kt:102) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$interceptCall$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:91) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:194) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.features.StatusPages.interceptCall(StatusPages.kt:101) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invokeSuspend(StatusPages.kt:142) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.StatusPages$Feature$install$2.invoke(StatusPages.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invokeSuspend(CallLogging.kt:139) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.features.CallLogging$Feature$install$2.invoke(CallLogging.kt) [ktor-server-core-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:118) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt) [ktor-server-host-common-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77) [ktor-utils-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:118) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source) [kotlinx-coroutines-core-jvm-1.4.2.jar:?]
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:43) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:34) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:216) [ktor-server-netty-jvm-1.4.3.jar:1.4.3]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar:4.1.54.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

Can't override Kotlin module configuration using jackson dsl function

NettyApplicationEngine: providing a configureBootstrap in the configuration throws IllegalStateException: group set already

Other

InvocationTargetException when running the default ktor project

https://youtrack.jetbrains.com/issue/KTOR-432

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1747

Hi all I've created the Ktor project in Intellj with Ktor plugin, when running the application it shows the following error message:

020-03-23 20:31:20.192 [main] INFO  Application - No ktor.deployment.watch patterns specified, automatic reload is not active
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Static.call(CallerImpl.kt:106)
	at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:166)
	at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:110)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.callFunctionWithInjection(ApplicationEngineEnvironmentReloading.kt:384)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:330)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$executeModuleFunction(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:275)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1$$special$$inlined$forEach$lambda$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:310)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.access$avoidingDoubleStartupFor(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:274)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading$instantiateAndConfigureApplication$1.invoke(ApplicationEngineEnvironmentReloading.kt:33)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.avoidingDoubleStartup(ApplicationEngineEnvironmentReloading.kt:290)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.instantiateAndConfigureApplication(ApplicationEngineEnvironmentReloading.kt:272)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.createApplication(ApplicationEngineEnvironmentReloading.kt:125)
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.start(ApplicationEngineEnvironmentReloading.kt:245)

The java version, that is installed on my computer:

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (Zulu 8.44.0.11-CA-linux64) (build 1.8.0_242-b20)
OpenJDK 64-Bit Server VM (Zulu 8.44.0.11-CA-linux64) (build 25.242-b20, mixed mode)

And here in the Application.kt:

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)

@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(Authentication) {
    }

    install(ContentNegotiation) {
        gson {
        }
    }

    val client = HttpClient() {
        install(JsonFeature) {
            serializer = GsonSerializer()
        }
        install(Logging) {
            level = LogLevel.HEADERS
        }
        BrowserUserAgent() // install default browser-like user-agent
        // install(UserAgent) { agent = "some user agent" }
    }
    runBlocking {
        // Sample for making a HTTP Client request
        /*
        val message = client.post<JsonSampleClass> {
            url("http://127.0.0.1:8080/path/to/endpoint")
            contentType(ContentType.Application.Json)
            body = JsonSampleClass(hello = "world")
        }
        */
    }

    routing {
        get("/") {
            call.respondText("HELLO WORLD!", contentType = ContentType.Text.Plain)
        }

        get("/json/gson") {
            call.respond(mapOf("hello" to "world"))
        }
    }
}

data class JsonSampleClass(val hello: String)

Thanks

Expose LocationInfo in ktor locations

Reload does not work with "user-local default: JAR manifest" command line shortener

HttpClient has strange behaviour when made many parellels requests (JsonFeature with GsonSerializer enabled)

https://youtrack.jetbrains.com/issue/KTOR-708

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/821

Hi Ktor team!

I have some problems with the client when make many parallels http requests.

I hope logs and code examples will help you.

Ktor Version

1.0.1

Ktor Engine Used

Android Engine

Features enabled

JsonFeature with GsonSerializer

Platform and hardware

Android 7, 4-core ARM CPU

Feedback

fun test() {
    for (i in 1..100) {
        println("Start launch")
        GlobalScope.launch(Dispatchers.Default) {
            val data = client.get<MyResponse>(url)
            println("Response completed: $i")
        }
        println("Finish launch")
    }
}`

Logs (please see time, it important, in really one request take about 1-3 seconds): ` 2018-12-24 01:07:47.388 I/System.out: Start launch 2018-12-24 01:07:47.403 I/System.out: Finish launch

// skiped logs: started 100 coroutines

2018-12-24 01:07:47.487 I/System.out: Start launch 2018-12-24 01:07:47.488 I/System.out: Finish launch // last coroutine started

// there is a big pause: in really one request take about 1-3 seconds

2018-12-24 01:07:57.671 I/System.out: Response completed: 9 // first response

// there is a big pause: in really request take about 1-3 seconds

2018-12-24 01:08:27.280 I/System.out: Response completed: 94 // second response 2018-12-24 01:08:28.265 I/System.out: Response completed: 1 2018-12-24 01:08:28.275 I/System.out: Response completed: 2 2018-12-24 01:08:28.285 I/System.out: Response completed: 4 2018-12-24 01:08:28.310 I/System.out: Response completed: 8 2018-12-24 01:08:28.396 I/System.out: Response completed: 80

// skiped logs: responses are distributed evenly in time

2018-12-24 01:08:29.268 I/System.out: Response completed: 92 2018-12-24 01:08:29.280 I/System.out: Response completed: 97 2018-12-24 01:08:29.296 I/System.out: Response completed: 99 2018-12-24 01:08:29.371 I/System.out: Response completed: 100 // last response`

I overrided GsonSerializer and added logs into problematic code.

override suspend fun read(type: TypeInfo, response: HttpResponse): Any {
    println("Start response.readText on ${Thread.currentThread().name}")
    val text = response.readText()
    println("End response.readText on ${Thread.currentThread().name}")
    return backend.fromJson(text, type.reifiedType) ?: Any()
}

Logs:

`2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-3 // first start readText 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:32:38.895 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:32:40.160 I/System.out: Start response.readText on DefaultDispatcher-worker-1

// skiped 45 logs Start response.readText on ..

2018-12-24 01:33:11.137 I/System.out: End response.readText on DefaultDispatcher-worker-3 // first end readText 2018-12-24 01:33:11.311 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:12.202 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:12.298 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:12.663 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:12.668 I/System.out: Start response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:14.052 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:14.091 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:14.238 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:14.313 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.475 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.486 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.576 I/System.out: Start response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.646 I/System.out: Start response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:15.746 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.754 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.766 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:15.877 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:15.914 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:15.930 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:15.938 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:15.995 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:16.051 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:16.101 I/System.out: End response.readText on DefaultDispatcher-worker-2

// skiped many End response.readText on ..

2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-4 2018-12-24 01:33:16.963 I/System.out: End response.readText on DefaultDispatcher-worker-3 2018-12-24 01:33:16.974 I/System.out: End response.readText on DefaultDispatcher-worker-1 2018-12-24 01:33:16.982 I/System.out: End response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:17.115 I/System.out: Start response.readText on DefaultDispatcher-worker-2 2018-12-24 01:33:17.129 I/System.out: End response.readText on DefaultDispatcher-worker-3 // last end readText`

I think problem in content.readRemaining method.

Maybe related https://github.com/ktorio/ktor/issues/768, https://github.com/ktorio/ktor/issues/770

I am ready to provide more details let me know.

Thanks!

InsufficientSpaceException trying to build ByteReadPacket

Release 1.5.3

Fix flaky ExceptionsJvmTest.testConnectionClosedDuringRequest[jvm]

Fix flaky JettyStressTest.highLoadStressTest

Fix flaky ProxyTest.testHttpProxy[CIO][jvm]

Fix flaky JavaEngineTests.testThreadLeak[jvm]

Incorrect grammar in exception messages

Invalid assertion for existence of the key in the key store

Ktor 1.5.3 relies on kotlinx-html-jvm:0.7.3

1.5.2

released 26th February 2021

Client

JS Fetch API switch to Standard Library (org.w3c.fetch.*)

HTTP Client exception is masked by JobCancellationException with Ktor 1.5.0

https://youtrack.jetbrains.com/issue/KTOR-1967

STR: perform http GET with ktor Apache client to a resource which replies with truncated content

AR:

kotlinx.coroutines.JobCancellationException: Parent job is Cancelling; job=JobImpl{Cancelled}@eb5ab92
	at kotlinx.coroutines.JobSupport.getChildJobCancellationCause(JobSupport.kt:711)
	at kotlinx.coroutines.JobSupport.createCauseException(JobSupport.kt:717)
	at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:692)
	at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
	at kotlinx.coroutines.JobSupport.parentCancelled(JobSupport.kt:634)
	at kotlinx.coroutines.ChildHandleNode.invoke(JobSupport.kt:1465)
	at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:1500)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
	at kotlinx.coroutines.JobSupport.cancelMakeCompleting(JobSupport.kt:693)
	at kotlinx.coroutines.JobSupport.cancelImpl$kotlinx_coroutines_core(JobSupport.kt:664)
	at kotlinx.coroutines.JobSupport.childCancelled(JobSupport.kt:648)
	at kotlinx.coroutines.ChildHandleNode.childCancelled(JobSupport.kt:1466)
	at kotlinx.coroutines.JobSupport.cancelParent(JobSupport.kt:358)
	at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
	at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:897)
	at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:860)
	at kotlinx.coroutines.JobSupport.makeCompleting$kotlinx_coroutines_core(JobSupport.kt:803)
	at kotlinx.coroutines.JobImpl.completeExceptionally(JobSupport.kt:1326)
	at io.ktor.client.engine.apache.ApacheRequestProducer.failed(ApacheRequestProducer.kt:72)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:98)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:276)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	... 1 more
Caused by: org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
	at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
	at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
	at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
	at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	... 10 more

ER: ConnectionClosedException is not masked

org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 178; received: 0)
	at org.apache.http.impl.nio.codecs.LengthDelimitedDecoder.read(LengthDelimitedDecoder.java:89)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:52)
	at io.ktor.client.engine.apache.ApacheResponseConsumer$consumeContent$1.invoke(ApacheResponseConsumer.kt:18)
	at io.ktor.utils.io.ByteBufferChannel.writeAvailable(ByteBufferChannel.kt:1631)
	at io.ktor.utils.io.ByteWriteChannel$DefaultImpls.writeAvailable$default(ByteWriteChannel.kt:85)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.consumeContent(ApacheResponseConsumer.kt:51)
	at org.apache.http.impl.nio.client.MainClientExec.consumeContent(MainClientExec.java:329)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.consumeContent(DefaultClientExchangeHandlerImpl.java:157)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:336)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)

Changing `requestTimeoutMillis` in config of HttpTimeout feature doesn't change the CIO's timeout

Core

Session cookie with very long max age duration

Ktor Client CIO engine Jvm ignores Ciphersuites with key strength more than 128 bits.

Docs

[Outdated doc] Ktor quickstart contains outdated reference to version before 1.0

Custom response validation is not running when default is disabled

Docker doc is incorrect / does not work

Documentation search doesn't work on the website

Plugin

Plugin does not generate a route for CSS DSL

Wizard: Markdown Preview throws exceptions

https://youtrack.jetbrains.com/issue/KTOR-2297

Occurred 10 times in the last 5 or so minutes, just doing basic scrolling and picking and choosing of features.

java.lang.NoSuchMethodError: 'void org.intellij.plugins.markdown.ui.preview.MarkdownHtmlPanel.render()'
	at io.ktor.initializr.intellij.features.KtorFeatureDetailsPageComponent$showKtorFeature$3$1.run(KtorFeatureDetailsPageComponent.kt:263)
	at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313)
	at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:776)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727)
	at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85)
	at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:746)
	at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:969)
	at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:839)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:449)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:808)
	at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$9(IdeEventQueue.java:448)
	at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:781)
	at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:502)
	at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124)
	at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109)
	at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
	at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90)

IntelliJ IDEA 2021.1 Beta (Ultimate Edition)
Build #IU-211.6305.21, built on March 4, 2021
IntelliJ IDEA EAP User
Expiration date: April 4, 2021
Runtime version: 11.0.10+9-b1341.18 x86_64
VM: Dynamic Code Evolution 64-Bit Server VM by JetBrains s.r.o.
macOS 10.14.6
GC: G1 Young Generation, G1 Old Generation
Memory: 4096M
Cores: 8
Registry: ide.images.show.chessboard=true
Non-Bundled Plugins: com.jetbrains.darkPurpleTheme (1.2), org.nik.presentation-assistant (1.0.9), wu.seal.tool.jsontokotlin (3.6.1), org.asciidoctor.intellij.asciidoc (0.32.26), stardust (1.5.2105), com.jetbrains.intellij.api.watcher (6.46.0), intellij.ktor (1.5.1-eap-1)
Kotlin: 211-1.4.21-release-IJ6305.1

Outdated Kotlin version in Ktor plugin

Samples

ktor-samples/client-mpp doesn't build client-mpp-ios in Android Studio

ktor-samples/client-mpp Android Studio doesn't see imports in iosMain

Server

CIO server always start on "0.0.0.0" - does not respect "connector" configuration

Digest authentication: cannot successfully pass authentication using curl or web browser

https://youtrack.jetbrains.com/issue/KTOR-1466

To reproduce run the server using the following code snippet:

fun main() {
    embeddedServer(Netty, port = 8081) {
        install(Authentication) {
            digest("digest") {
                val password = "123"
                realm = "testrealm@host.com"
                digestProvider { name, realm ->
                    "$name:$realm:$password".toByteArray()
                }
            }
        }
        routing {
            authenticate("digest") {
                get("/digest") {
                    call.respondText("ok")
                }
            }
        }
    }.start(wait = true)
}

Send the following HTTP request using curl:

curl -v  'http://localhost:8081/digest' --digest -u guest:123

I expect successfully pass authentication and get a response with a status code 200 but instead, I get the following response:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="43f48979ca35e2a5", algorithm="MD5"
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8081/digest'
* Found bundle for host localhost: 0x7fbe62518750 [can pipeline]
* Could pipeline, but not asked to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8081 (#0)
* Server auth using Digest with user 'guest'
> GET /digest HTTP/1.1
> Host: localhost:8081
> Authorization: Digest username="guest", realm="testrealm@host.com", nonce="43f48979ca35e2a5", uri="/digest", response="ee8c56f444996c77cd1e5f7570035372", algorithm="MD5"
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="testrealm@host.com", nonce="76d13a440ec99b1c", algorithm="MD5"
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

server/netty: IllegalReferenceCountException

https://youtrack.jetbrains.com/issue/KTOR-1801

I got this random error out of nowhere. I'm not even using the Ktor server. It's just random bots on the web making HTTP calls to it. I have no idea what request, call or response it was.

2021-01-19 21:21:41.833 E Unhandled exception caught for CoroutineName(call-handler)
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
	at io.netty.handler.codec.http.DefaultFullHttpRequest.release(DefaultFullHttpRequest.java:103)
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
	at io.ktor.server.netty.NettyApplicationCall.finishComplete(NettyApplicationCall.kt:53)
	at io.ktor.server.netty.NettyApplicationCall.finishSuspend(NettyApplicationCall.kt:46)
	at io.ktor.server.netty.NettyApplicationCall$finishSuspend$1.invokeSuspend(NettyApplicationCall.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:227)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:832)

Installed features: CallLogging, Compression, DefaultHeaders, CORS, XForwardedHeaderSupport, WebSockets (unused)

Routes: 1x get, 1x post (both for GraphQL queries)

Mandatory Path Segment parameter can be empty, if no explicit route with trailing / is defined

Other

Test a POST with MultiPart using TestApplicationEngine does not success or fail

https://youtrack.jetbrains.com/issue/KTOR-345

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1936

Ktor 1.3.2, TestApplicationEngine

When running a test that sends post with multipart it runs forever and never stops.

No problem when server really runs. It's just the test.

To Reproduce

Server:

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)

@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(ContentNegotiation) {
    }

    routing {
        post("/") {
            val mp = try {
                call.receiveMultipart()
            } catch (e: Exception) {
                call.application.log.error("Getting multipart error"")
                null
            }

           if (mp == null) call.respond(HttpStatusCode.BadRequest, "Null value")
           else call.respond(HttpStatusCode.OK,"OK")
        }
    }
}

Test:

internal class ApplicationKtTest {
    private val boundary = "***bbb***"
    private val  sysTempDir = System.getProperty("java.io.tmpdir") ?: "/tmp"

    private val multipart = listOf(PartData.FileItem({ byteArrayOf(1, 2, 3).inputStream().asInput() }, {}, headersOf(
            HttpHeaders.ContentDisposition,
            ContentDisposition.File
                    .withParameter(ContentDisposition.Parameters.Name, "file")
                    .withParameter(ContentDisposition.Parameters.FileName, "test.jpg")
                    .toString()
    )))

    @Test
    fun testUploadApplication() = testApp {
        handlePost("/", boundary, multipart).apply {
            println("============= RESPONSE ====================")
            println(response.content)
            println("=================================")
        }
    }
}

private fun TestApplicationEngine.handlePost(uri: String,
         boundary: String,
         multipart: List<PartData.FileItem>,
         setup: TestApplicationRequest.() -> Unit = {}
): TestApplicationCall {
    return handleRequest(method = HttpMethod.Post, uri = uri)  {
        addHeader(HttpHeaders.ContentType,
                ContentType.MultiPart.FormData.withParameter("boundary", boundary).toString()
        )
        setBody(boundary, multipart)
        setup()
    }
}

private fun testApp(callback: TestApplicationEngine.() -> Unit): Unit {
    withTestApplication({ module(true) }, callback)
}

The test runs forever and gives no result. if addHeader and setBody are commented there is a response.

Expected behavior The test should print OK

Repository If you just want to clone a test repository.

CIO Server Engine fails for requests with more than 32 headers

Crash with Firebase Performance in iOS

Release 1.5.2

Default Headers feature adds duplicated Server header

Fix flaky ClientSocketTest.testSelfConnect[jvm]

Fix testUnsafeHeaders for Js platform

Native does not build on linux machine

Fix Dokka building for master

Fix existing samples

1.5.1

released 28th January 2021

Client

Remove dependency on 'text-encoding' npm package

Jetty: requests to resources, that doesn't respond with HTTP/2, lead to unexpected behaviour

https://youtrack.jetbrains.com/issue/KTOR-874

To reproduce run the following code:

fun main() {
    val client = HttpClient(Jetty) {}
    runBlocking {
        val r = client.get<String>("http://www.google.com")
        println(r)
    }
}

As a result, org.eclipse.jetty.io.EofException will be thrown:

Exception in thread "main" org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:259)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
	at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:543)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt:40)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$jettyRequest$1.invoke(JettyHttpRequest.kt)
	at io.ktor.client.engine.jetty.UtilsKt.withPromise(utils.kt:14)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt.executeRequest(JettyHttpRequest.kt:39)
	at io.ktor.client.engine.jetty.JettyHttpRequestKt$executeRequest$1.invokeSuspend(JettyHttpRequest.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: java.nio.channels.AsynchronousCloseException
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
	... 18 more

If I change the URL to http://example.com then the app will hang indefinitely, here is a stacktrace:

"main" #1 prio=5 os_prio=0 cpu=337.00ms elapsed=56.16s tid=0x00007f8e98018800 nid=0x79547 waiting on condition  [0x00007f8e9eb77000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x00000006890f1518> (a kotlinx.coroutines.BlockingCoroutine)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:87)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at MainKt.main(main.kt:86)
	at MainKt.main(main.kt)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0.36ms elapsed=56.11s tid=0x00007f8e98311000 nid=0x79550 waiting on condition  [0x00007f8e80382000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.7/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.7/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.7/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=1.63ms elapsed=56.11s tid=0x00007f8e98315000 nid=0x79551 in Object.wait()  [0x00007f8e80281000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.7/Native Method)
	- waiting on <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000006890f2008> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.7/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0.22ms elapsed=56.10s tid=0x00007f8e9832a000 nid=0x79552 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 cpu=497.80ms elapsed=56.10s tid=0x00007f8e9832c000 nid=0x79553 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=306.59ms elapsed=56.10s tid=0x00007f8e9832e000 nid=0x79554 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #7 daemon prio=9 os_prio=0 cpu=6.70ms elapsed=56.09s tid=0x00007f8e98330000 nid=0x79555 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #8 daemon prio=8 os_prio=0 cpu=0.37ms elapsed=56.03s tid=0x00007f8e98381800 nid=0x79556 in Object.wait()  [0x00007f8e47efd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.7/Native Method)
	- waiting on <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000006890f27e0> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.7/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.7/InnocuousThread.java:134)

"Monitor Ctrl-Break" #9 daemon prio=5 os_prio=0 cpu=8.39ms elapsed=55.91s tid=0x00007f8e98695800 nid=0x79557 runnable  [0x00007f8e479d4000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.7/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.7/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.7/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.7/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.7/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.7/StreamDecoder.java:178)
	- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.7/InputStreamReader.java:185)
	at java.io.BufferedReader.fill(java.base@11.0.7/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:326)
	- locked <0x00000006890f4d00> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.7/BufferedReader.java:392)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:61)

"Service Thread" #10 daemon prio=9 os_prio=0 cpu=0.04ms elapsed=55.91s tid=0x00007f8e98696800 nid=0x79558 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"ktor-jetty-dispatcher-worker-1" #11 daemon prio=5 os_prio=0 cpu=230.54ms elapsed=55.68s tid=0x00007f8e989de000 nid=0x7955a waiting on condition  [0x00007f8e46ec0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"ktor-jetty-dispatcher-worker-2" #12 daemon prio=5 os_prio=0 cpu=9.27ms elapsed=55.68s tid=0x00007f8e989df000 nid=0x7955b waiting on condition  [0x00007f8e46dbf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:357)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.park(CoroutineScheduler.kt:783)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.tryPark(CoroutineScheduler.kt:728)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:711)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"ktor-jetty-client-qtp-13" #13 prio=5 os_prio=0 cpu=6.95ms elapsed=55.47s tid=0x00007f8e340da000 nid=0x7955d runnable  [0x00007f8e46575000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPoll.wait(java.base@11.0.7/Native Method)
	at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.7/EPollSelectorImpl.java:120)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.7/SelectorImpl.java:124)
	- locked <0x000000069f414300> (a sun.nio.ch.Util$2)
	- locked <0x000000069f4142a8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(java.base@11.0.7/SelectorImpl.java:141)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:472)
	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:409)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:360)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:184)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.io.ManagedSelector$$Lambda$65/0x0000000800176c40.run(Unknown Source)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-14" #14 prio=5 os_prio=0 cpu=126.50ms elapsed=55.47s tid=0x00007f8e340dc000 nid=0x7955e waiting on condition  [0x00007f8e46474000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-15" #15 prio=5 os_prio=0 cpu=0.27ms elapsed=55.47s tid=0x00007f8e340dd800 nid=0x7955f waiting on condition  [0x00007f8e46373000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069fc076c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2211)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:309)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:379)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-16" #16 prio=5 os_prio=0 cpu=2.68ms elapsed=55.46s tid=0x00007f8e340df800 nid=0x79560 waiting on condition  [0x00007f8e46272000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-17" #17 prio=5 os_prio=0 cpu=0.10ms elapsed=55.46s tid=0x00007f8e340e1000 nid=0x79561 waiting on condition  [0x00007f8e46171000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-18" #18 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e3000 nid=0x79562 waiting on condition  [0x00007f8e46070000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-19" #19 prio=5 os_prio=0 cpu=0.07ms elapsed=55.46s tid=0x00007f8e340e5000 nid=0x79563 waiting on condition  [0x00007f8e45f6f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"ktor-jetty-client-qtp-20" #20 prio=5 os_prio=0 cpu=0.06ms elapsed=55.46s tid=0x00007f8e340e7000 nid=0x79564 waiting on condition  [0x00007f8e45e6e000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f418170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.7/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.7/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"Scheduler-779376001-1" #21 prio=5 os_prio=0 cpu=10.86ms elapsed=55.42s tid=0x00007f8e34117000 nid=0x79565 waiting on condition  [0x00007f8e45b6d000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x000000069f415c20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.7/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:1170)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.7/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.7/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

"Attach Listener" #22 daemon prio=9 os_prio=0 cpu=8.56ms elapsed=55.08s tid=0x00007f8e4805c800 nid=0x79567 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=16.20ms elapsed=56.11s tid=0x00007f8e98309000 nid=0x7954f runnable  

"GC Thread#0" os_prio=0 cpu=22.88ms elapsed=56.16s tid=0x00007f8e98041800 nid=0x7954a runnable  

"GC Thread#1" os_prio=0 cpu=22.20ms elapsed=55.66s tid=0x00007f8e5c001000 nid=0x7955c runnable  

"G1 Main Marker" os_prio=0 cpu=0.51ms elapsed=56.15s tid=0x00007f8e980cc800 nid=0x7954b runnable  

"G1 Conc#0" os_prio=0 cpu=0.07ms elapsed=56.14s tid=0x00007f8e980ce000 nid=0x7954c runnable  

"G1 Refine#0" os_prio=0 cpu=0.50ms elapsed=56.13s tid=0x00007f8e9823b800 nid=0x7954d runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=10.66ms elapsed=56.13s tid=0x00007f8e9823d800 nid=0x7954e runnable  
"VM Periodic Task Thread" os_prio=0 cpu=42.21ms elapsed=55.91s tid=0x00007f8e98698000 nid=0x79559 waiting on condition  

JNI global refs: 14, weak refs: 0

If I make the same requests with curl: curl --http2-prior-knowledge http://example.com then I get an understandable error:

curl: (16) Error in the HTTP2 framing layer

Circular reference for SocketException and StackOverflowError when using SLF4J logger

https://youtrack.jetbrains.com/issue/KTOR-1080

I upgraded my project to ktor 1.4.1 today. I found an issue when I ran my unit test. My logger failed to log the SocketTimeoutException.

Sample code:

fun main() {
    val httpClient = HttpClient(Apache) {
        install(HttpTimeout) {
            socketTimeoutMillis = 3000
        }
    }

    runBlocking {
        try {
            val get = httpClient.get<String>("https://httpbin.org/delay/10")
        } catch (e: Exception) {
            e.printStackTrace()
            LoggerFactory.getLogger("main").error("test", e)
        }
    }
}

Output:

io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
	at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.sendRequest(ApacheHttpRequest.kt:49)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt$sendRequest$1.invokeSuspend(ApacheHttpRequest.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Caused by: io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms
	at io.ktor.client.features.HttpTimeoutKt.SocketTimeoutException(HttpTimeout.kt:177)
	at io.ktor.client.engine.apache.ApacheHttpRequestKt.mapCause(ApacheHttpRequest.kt:57)
	at io.ktor.client.engine.apache.ApacheResponseConsumer.failed(ApacheResponseConsumer.kt:73)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:99)
	at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:834)
	[CIRCULAR REFERENCE:io.ktor.network.sockets.SocketTimeoutException: Socket timeout has been expired [url=https://httpbin.org/delay/10, socket_timeout=3000] ms]
Caused by: java.net.SocketTimeoutException: 3,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
	... 11 more
Exception in thread "main" java.lang.StackOverflowError
	at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:590)
	at java.base/java.lang.reflect.Method.invoke(Method.java:558)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:66)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:60)
	at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:72)

"Unfinished workers detected" using client on native

OkHTTP client engine tries to close the connection twice during the closing handshake

https://youtrack.jetbrains.com/issue/KTOR-1374

Here's how I noticed:

When I cancel the coroutine containing the WebSocket connection, KTor closes the connection with a 1011. If the server then responds by closing the connection without a status code, KTor will get a 1005 and it'll try to pass it to OkHTTP.

However, 1005 is a reserved status, so OkHTTP refuses it when Ktor passes it in the following lines: https://github.com/ktorio/ktor/blob/6053fa44a61fdfc5474ff909c0224e56877b5da1/ktor-client/ktor-client-okhttp/jvm/src/io/ktor/client/engine/okhttp/OkHttpWebsocketSession.kt#L62

Here's an example of a traceback you'd get:

    java.util.concurrent.CancellationException: ActorCoroutine was cancelled
        at kotlinx.coroutines.ExceptionsKt.CancellationException(Exceptions.kt:22)
        at kotlinx.coroutines.channels.ActorCoroutine.onCancelling(Actor.kt:134)
        at kotlinx.coroutines.JobSupport.notifyCancelling(JobSupport.kt:332)
        at kotlinx.coroutines.JobSupport.tryMakeCompletingSlowPath(JobSupport.kt:916)
        at kotlinx.coroutines.JobSupport.tryMakeCompleting(JobSupport.kt:875)
        at kotlinx.coroutines.JobSupport.makeCompletingOnce$kotlinx_coroutines_core(JobSupport.kt:840)
        at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:111)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
     Caused by: java.lang.IllegalArgumentException: Code 1005 is reserved and may not be used.
        at okhttp3.internal.ws.WebSocketProtocol.validateCloseCode(WebSocketProtocol.kt:134)
        at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:435)
        at okhttp3.internal.ws.RealWebSocket.close(RealWebSocket.kt:427)
        at io.ktor.client.engine.okhttp.OkHttpWebsocketSession$outgoing$1.invokeSuspend(OkHttpWebsocketSession.kt:62)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) 
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

I think there are technically two different issues happening at the same time:

Ktor is trying to close the connection twice. The first time with a 1011 status (which I can confirm in the server logs) and the second time with a 1005 (because the server didn't provide a closing status code).
Ktor is trying to close the connection with a reserved status code.

ClosedReceiveChannelException when making request with CIO engine using a proxy to https

https://youtrack.jetbrains.com/issue/KTOR-1458

version: 1.4.3

when use a http proxy, get the exception.

use no proxy, it's ok.

code


val httpClient = HttpClient(CIO) {
    engine {
        proxy = httpProxy
    }
}

val c : String = httpClient.get(url)

build.gradle.kts

// ...

dependencies {
    implementation("org.jetbrains.kotlinx", "kotlinx-serialization-json", "1.0.1")
    implementation("io.ktor", "ktor-client-core", "1.4.3")
    implementation("io.ktor", "ktor-client-cio", "1.4.3")
    implementation("io.ktor", "ktor-client-serialization-jvm", "1.4.3")
    implementation("org.jetbrains.kotlinx", "kotlinx-coroutines-core", "1.4.2")

    // test
}

exception:

Channel was closed
kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	(Coroutine boundary)
	at kotlinx.coroutines.CompletableDeferredImpl.await(CompletableDeferred.kt:86)
	at io.ktor.client.engine.cio.Endpoint.execute(Endpoint.kt:76)
	at io.ktor.client.engine.cio.CIOEngine.execute(CIOEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:86)
	(Coroutine creation stacktrace)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:30)
	at kotlinx.coroutines.BuildersKt.async$default(Unknown Source)
	at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:81)
	at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:66)
	at io.ktor.client.engine.HttpClientEngine$install$1.invoke(HttpClientEngine.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:128)
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:92)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:123)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at LoginTest$testLogin$1.invokeSuspend(LoginTest.kt:18)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at mytest(...)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
	at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:133)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at kotlinx.coroutines.DeferredCoroutine.await$suspendImpl(Builders.common.kt:101)
	at io.ktor.client.features.HttpSend$DefaultSender.execute(HttpSend.kt:129)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:188)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at myfunction(...)
	at mytest(...)
Caused by: kotlinx.coroutines.channels.ClosedReceiveChannelException: Channel was closed
	at kotlinx.coroutines.channels.Closed.getReceiveException(AbstractChannel.kt:1107)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveElement.resumeReceiveClosed(AbstractChannel.kt:906)
	at kotlinx.coroutines.channels.AbstractSendChannel.helpClose(AbstractChannel.kt:322)
	at kotlinx.coroutines.channels.AbstractSendChannel.close(AbstractChannel.kt:251)
	at kotlinx.coroutines.channels.ChannelCoroutine.close(ChannelCoroutine.kt)
	at kotlinx.coroutines.channels.SendChannel$DefaultImpls.close$default(Channel.kt:105)
	at io.ktor.network.tls.TLSClientHandshake$input$1.invokeSuspend(TLSClientHandshake.kt:87)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

ktor-client-apache: thread stuck in ByteBufferChannel.readRemainingSuspend

https://youtrack.jetbrains.com/issue/KTOR-1463

In production I've got two threads stuck in 100% CPU usage due to Ktor.

The server is running without debugging agent, so I cannot debug the exact scenario just yet. I'll restart it today with debugging agent running so that I can investigate further the next time the issue occurs.

When creating thread dumps I get one of the following two outputs. Note the high CPU usage and that the threads seem to be stuck in ByteBufferChannel.readRemainingSuspend.

"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51203112.18ms elapsed=53179.31s tid=0x00007f6fc0052510 nid=0xa9b2 runnable  [0x00007f6f0ebe9000]
   java.lang.Thread.State: RUNNABLE
	at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible$default(ByteBufferChannel.kt:2678)
	at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:2231)
	at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

"DefaultDispatcher-worker-2" #423 daemon prio=5 os_prio=0 cpu=51253887.85ms elapsed=53230.30s tid=0x00007f6fc0052510 nid=0xa9b2 runnable  [0x00007f6f0ebe9000]
   java.lang.Thread.State: RUNNABLE
	at io.ktor.utils.io.core.internal.UnsafeKt.prepareWriteHead(Unsafe.kt:160)
	at io.ktor.utils.io.ByteBufferChannel.readRemainingSuspend(ByteBufferChannel.kt:3060)
	at io.ktor.utils.io.ByteBufferChannel$readRemainingSuspend$1.invokeSuspend(ByteBufferChannel.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

ResponseException is no longer serializable starting from 1.4.0 (breaking change)

https://youtrack.jetbrains.com/issue/KTOR-1552

Previously a ResponseException could be serialized. See: https://github.com/ktorio/ktor/issues/1256. The Transient annotation was removed in a recent commit for the HttpResponse field. Now an exception is thrown when serializing a ResponseException.

Stack trace:

io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
java.io.NotSerializableException: io.ktor.utils.io.concurrent.SharedJvmKt$threadLocal$1
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
	at java.base/java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
	at java.base/java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
	at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
	at ExceptionsTest.serialize(ExceptionsTest.kt:26)
	at ExceptionsTest.testResponseExceptionSerializable(ExceptionsTest.kt:17)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)

Here is a test to reproduce the issue:

import io.ktor.client.call.*
import io.ktor.client.features.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.util.date.*
import io.ktor.utils.io.*
import java.io.*
import kotlin.coroutines.*
import kotlin.test.*

class ExceptionsTest {

    @Test
    fun testResponseExceptionSerializable() {
        val exception = createResponseException()

        val serialized = serialize(exception)
        val deserialized = deserialize(serialized)

        deserialized as ResponseException
    }

    private fun serialize(obj: Any): ByteArray {
        val b = ByteArrayOutputStream()
        val o = ObjectOutputStream(b)
        o.writeObject(obj)
        return b.toByteArray()
    }

    private fun deserialize(bytes: ByteArray): Any? {
        val b = ByteArrayInputStream(bytes)
        val o = ObjectInputStream(b)
        return o.readObject()
    }
}

private fun createResponseException(): ResponseException = ResponseException(object : HttpResponse() {
    override val call: HttpClientCall
        get() = TODO("Not yet implemented")
    override val status: HttpStatusCode
        get() = TODO("Not yet implemented")
    override val version: HttpProtocolVersion
        get() = TODO("Not yet implemented")
    override val requestTime: GMTDate
        get() = TODO("Not yet implemented")
    override val responseTime: GMTDate
        get() = TODO("Not yet implemented")
    override val content: ByteReadChannel
        get() = TODO("Not yet implemented")
    override val headers: Headers
        get() = TODO("Not yet implemented")
    override val coroutineContext: CoroutineContext
        get() = TODO("Not yet implemented")

    override fun toString(): String = "FakeCall"
}, cachedResponseText = "Fake text")

Java client freeze

HttpTimeoutTest.testConnect are flaky

HttpTimeout.testSocketTimeoutWriteFail is flaky

Response channel is always cancelled with Logging feature

Java client logging tests are fluky

CIO native selector doesn't select new descriptors

Logging tests fails due to floating log entries

Core

Unable to catch socket exceptions

https://youtrack.jetbrains.com/issue/KTOR-1166

Exceptions thrown from the UDP sockets cannot be caught.

Here is a test that currently fails:

import io.ktor.network.selector.*
import io.ktor.network.sockets.*
import io.ktor.util.network.*
import io.ktor.utils.io.core.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import kotlin.io.use
import kotlin.test.*

class SocketExceptionIssue : CoroutineScope {
    private val testJob = Job()
    private val selector = ActorSelectorManager(Dispatchers.Default + testJob)

    override val coroutineContext: CoroutineContext
        get() = testJob

    @AfterTest
    fun tearDown() {
        testJob.cancel()
        selector.close()
    }

    @Test
    fun testBroadcastFails(): Unit = runBlocking {
        withTimeout(15000) {
            // The following code in the assertFails lambda should throw a SocketException.
            assertFails {
                aSocket(selector)
                    .udp()
                    .bind {
                        // Do not set broadcast to true so exception is thrown.
                    }
                    .use { socket ->
                        // Send a broadcast which will fail because broadcasts are not enabled on this socket.
                        socket.send(
                            Datagram(
                                packet = buildPacket { writeText("0123456789") },
                                address = NetworkAddress("255.255.255.255", 56700)
                            )
                        )
                    }
            }
        }
    }
}

When running this test the logs show the following:

> Task :ktor-network:jvmTest
Exception in thread "DefaultDispatcher-worker-1 @coroutine#3" java.net.SocketException: Permission denied
	at java.base/sun.nio.ch.DatagramChannelImpl.send0(Native Method)
	at java.base/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:584)
	at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:546)
	at java.base/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:529)
	at io.ktor.network.sockets.DatagramSocketImpl.sendImpl(DatagramSocketImpl.kt:98)
	at io.ktor.network.sockets.DatagramSocketImpl$sender$1.invokeSuspend(DatagramSocketImpl.kt:35)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

Expected an exception to be thrown, but was completed successfully.
java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
	at org.junit.Assert.fail(Assert.java:88)
	at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
	at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
	at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
	at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	(Coroutine boundary)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	(Coroutine creation stacktrace)
	at kotlin.coroutines.intrinsics.IntrinsicsKt__IntrinsicsJvmKt.createCoroutineUnintercepted(IntrinsicsJvm.kt:122)
	at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable(Cancellable.kt:26)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue.testBroadcastFails(SocketExceptionIssue.kt:30)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.AssertionError: Expected an exception to be thrown, but was completed successfully.
	at org.junit.Assert.fail(Assert.java:88)
	at kotlin.test.junit.JUnitAsserter.fail(JUnitSupport.kt:56)
	at kotlin.test.AssertionsKt__AssertionsKt.checkResultIsFailure(Assertions.kt:153)
	at kotlin.test.AssertionsKt.checkResultIsFailure(Unknown Source)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invokeSuspend(SocketExceptionIssue.kt:33)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1$1.invoke(SocketExceptionIssue.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturnIgnoreTimeout(Undispatched.kt:102)
	at kotlinx.coroutines.TimeoutKt.setupTimeout(Timeout.kt:110)
	at kotlinx.coroutines.TimeoutKt.withTimeout(Timeout.kt:32)
	at io.ktor.network.sockets.tests.SocketExceptionIssue$testBroadcastFails$1.invokeSuspend(SocketExceptionIssue.kt:31)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:81)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:56)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt)
	... 50 more

io.ktor.network.sockets.tests.SocketExceptionIssue[jvm] > testBroadcastFails[jvm] FAILED
    java.lang.AssertionError at SocketExceptionIssue.kt:30
        Caused by: java.lang.AssertionError at SocketExceptionIssue.kt:30
1 test completed, 1 failed

You can see the SocketException in the log and that the test (with assertFails) is unable to catch it.

The following issue looks related: https://youtrack.jetbrains.com/issue/KTOR-440

Here is a fork of Ktor with the test added so you can reproduce the issue: https://github.com/Thomas-Vos/ktor/tree/socket_exception_issue (see last commit for added test). The fork is based on the 1.5.0 branch.

Incorrect encoding function used for URL path by URLBuilder

testTimeoutCancelsWhenParentScopeCancels is flaky

Upgrade kotlin to 1.4.21

Docs

"Using a Self-Signed Certificate" docs provide wrong dependency for 1.3.x

Out of date self-signed-certificate documentation

start.ktor.io - Incorrect import for websockets for ktor 1.2.4

Client logging docs don't mention all required dependencies

Custom JSON mapping with Jackson

Serialization for client section does not explain how to use it

https://youtrack.jetbrains.com/issue/KTOR-999

Serialization for client section does not explain how to use it and it doesn't tell anything about required headers. https://ktor.io/clients/http-client/features/json-feature.html -- This is the page that describes the Json feature. It only teaches how to install the feature (and reader assumes that it will work out of the box right after installation, and any data will be automatically serialized which is not quite true because it also requires contentType to be set).

This implies that user can just write something like:

@Serializable
data class HelloWorld(val text: String = "Hello world")

val client = HttpClient(Apache) {
    install(JsonFeature) {
        serializer = KotlinxSerializer()
    }
}

client.post<Unit> {
    url("http://127.0.0.1:8080/")
    body = HelloWorld()
}

But this is not true:

Exception in thread "main" java.lang.IllegalStateException: Fail to serialize body. Content has type: class HelloWorld, but OutgoingContent expected.
If you expect serialized body, please check that you have installed the corresponding feature(like `Json`) and set `Content-Type` header.
	at io.ktor.client.features.HttpSend$Feature$install$1.invokeSuspend(HttpSend.kt:79)
	at io.ktor.client.features.HttpSend$Feature$install$1.invoke(HttpSend.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(PipelineContext.kt:178)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:86)
	at io.ktor.client.features.HttpCallValidator$Companion$install$1.invoke(HttpCallValidator.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invokeSuspend(HttpRequestLifecycle.kt:37)
	at io.ktor.client.features.HttpRequestLifecycle$Feature$install$1.invoke(HttpRequestLifecycle.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:323)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(PipelineContext.kt:168)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(PipelineContext.kt:188)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:31)
	at io.ktor.client.HttpClient.execute(HttpClient.kt:185)
	at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:104)
	at MainKt$main$1.invokeSuspend(main.kt:49)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at MainKt.main(main.kt:19)

It also requires content type (ex., contentType(ContentType.Application.Json)) to be set. This information is not discoverable and should be documented.

"Testing Http Client" docs page contains artifact name with -native suffix

New documentation lacks artifacts information for Gradle and Maven

[docs] Wrong shadow plugin version in Fat JAR docs

Add information about required artifacts to the WebSockets topic

Missing dependency information the Client Auth topic

Missing dependency information the Authentication and Authorization topic

ktor server documentation is returning 404

Add "Edit on GitHub" button to docs

Plugin

Validate for Empty Input Fields in New Project Wizard

Server

Status-code must be 3-digit

Request parameters should have name

Reserved characters in path is not encoded

Autoreload embeddedServer configuration fails, if a module has default parameters

https://youtrack.jetbrains.com/issue/KTOR-1334

Ktor Version

1.4.2

Ktor Engine

Netty

JVM Version, Operating System and Relevant Context

JDK 11, Manjaro

Problem

Project is generated by Ktor IDEA plugin, which if test modules are included, generates a module with default parameters.
It fails with Module function provided as lambda cannot be unlinked for reload, if module has default parameters.
Using method reference as in the example, does not make it work.

Example:

fun main(args: Array<String>): Unit {
    embeddedServer(
            Netty,
            port = 8080,
            host = "0.0.0.0",
            watchPaths = listOf("some"),
            module = Application::module,
    ).start(true)
}

fun Application.module(testing: Boolean = false): Unit {
    routing {
        get("/") {
            call.respondText { "Hello" }
        }
    }
}

Running Result

Exception in thread "main" java.lang.RuntimeException: Module function provided as lambda cannot be unlinked for reload
	at io.ktor.server.engine.ApplicationEngineEnvironmentReloading.<init>(ApplicationEngineEnvironmentReloading.kt:58)
	at io.ktor.server.engine.ApplicationEngineEnvironmentBuilder.build(ApplicationEngineEnvironment.kt:105)
	at io.ktor.server.engine.ApplicationEngineEnvironmentKt.applicationEngineEnvironment(ApplicationEngineEnvironment.kt:44)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:55)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer(EmbeddedServer.kt:36)
	at io.ktor.server.engine.EmbeddedServerKt.embeddedServer$default(EmbeddedServer.kt:34)
	at some.ReloadKt.main(reload.kt:10)

It fails here, basically can't cast the default parameter method as KFunction<*>

public class ApplicationEngineEnvironmentReloading(
//...
) : ApplicationEngineEnvironment {
//...

    private val moduleFunctionNames: List<String>? = run {
        val configModules = config.propertyOrNull("ktor.application.modules")?.getList()
        if (watchPatterns.isEmpty()) configModules
        else {
            val unlinkedModules = modules.map {
         // Fails here
                val fn = (it as? KFunction<*>)?.javaMethod
                    ?: throw RuntimeException("Module function provided as lambda cannot be unlinked for reload")
                val clazz = fn.declaringClass
                val name = fn.name
                "${clazz.name}.$name"
            }
            if (configModules == null)
                unlinkedModules
            else
                configModules + unlinkedModules
        }
    //...
}

Workaround

fun Application.module() = module(false)

fun Application.module(testing: Boolean = false): Unit {...}

CallLoggingTest is flaky

Server losing channel exceptions at receive

CORS doesn't reject bad headers

Adding existing dropwizard metrics registry to Ktor

Other

Support explicit WebSocket session close

https://youtrack.jetbrains.com/issue/KTOR-340

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1946

Subsystem Client, and server possibly

Is your feature request related to a problem? Please describe.

    client.ws(host = "echo.websocket.org") {
        send(Frame.Text("Hello World"))
        close()

        incoming.consumeEach {
            println(it)
        }
    }

Fails with:

Exception in thread "main" java.util.concurrent.CancellationException: ArrayChannel was cancelled
	at kotlinx.coroutines.channels.AbstractChannel.cancel(AbstractChannel.kt:632)
	at kotlinx.coroutines.channels.ReceiveChannel$DefaultImpls.cancel$default(Channel.kt:311)
	at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl$runOutgoingProcessor$1.invokeSuspend(DefaultWebSocketSessionImpl.kt:163)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTaskKt.resume(DispatchedTask.kt:175)
	at kotlinx.coroutines.DispatchedTaskKt.resumeUnconfined(DispatchedTask.kt:137)
	at kotlinx.coroutines.DispatchedTaskKt.dispatch(DispatchedTask.kt:108)
	at kotlinx.coroutines.CancellableContinuationImpl.dispatchResume(CancellableContinuationImpl.kt:306)
	at kotlinx.coroutines.CancellableContinuationImpl.completeResume(CancellableContinuationImpl.kt:393)
	at kotlinx.coroutines.channels.AbstractChannel$ReceiveHasNext.completeResumeReceive(AbstractChannel.kt:918)
	at kotlinx.coroutines.channels.ArrayChannel.offerInternal(ArrayChannel.kt:83)
	at kotlinx.coroutines.channels.AbstractSendChannel.send(AbstractChannel.kt:149)
	at io.ktor.http.cio.websocket.WebSocketSession$DefaultImpls.send(WebSocketSessionJvm.kt:45)
	at io.ktor.http.cio.websocket.DefaultWebSocketSession$DefaultImpls.send(DefaultWebSocketSession.kt)
	at io.ktor.http.cio.websocket.DefaultWebSocketSessionImpl.send(DefaultWebSocketSessionImpl.kt:27)
	at io.ktor.client.features.websocket.DefaultClientWebSocketSession.send(ClientSessions.kt)
	at io.ktor.http.cio.websocket.WebSocketSessionKt.close(WebSocketSession.kt:74)
	at io.ktor.http.cio.websocket.WebSocketSessionKt.close$default(WebSocketSession.kt:72)
	at WebSocketKt$Websockets$1$1.invokeSuspend(WebSocket.kt:15)
	at WebSocketKt$Websockets$1$1.invoke(WebSocket.kt)
	at io.ktor.client.features.websocket.BuildersKt.webSocket(builders.kt:54)
	at io.ktor.client.features.websocket.BuildersKt$webSocket$1.invokeSuspend(builders.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at io.ktor.util.pipeline.SuspendFunctionGun.resumeRootWith(PipelineContext.kt:238)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(PipelineContext.kt:194)
	at io.ktor.util.pipeline.SuspendFunctionGun.access$loop(PipelineContext.kt:67)
	at io.ktor.util.pipeline.SuspendFunctionGun$continuation$1.resumeWith(PipelineContext.kt:144)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:272)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:79)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:54)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:36)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at WebSocketKt.Websockets(WebSocket.kt:7)
	at MainKt.main(main.kt:4)

Post request shows empty body after upgrading v1.3.2

https://youtrack.jetbrains.com/issue/KTOR-426

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1764

Ktor Version 1.3.2 - Android

Post request shows empty body

We are sending a post request using Ktor. After upgrading to V1.3.2 to support kotlin 1.3.7, the request body started showing empty and nothing is actually sent in the body. I used the debugger to trace it, I see that it is able to serialize my request object correctly and content type is done correctly with application/json, however at some point I see in the logs that it is nullified.

We are using JsonFeature and Logging feature and our event object has it's own serilaizer which I was able to see it successfully serialized inside Ktor client

This is from KotlinxSerializer.kt

    override fun write(data: Any, contentType: ContentType): OutgoingContent {
        @Suppress("UNCHECKED_CAST")
        val content = json.stringify(buildSerializer(data) as KSerializer<Any>, data)
        return TextContent(content, contentType)
    }

It is returning a valid TextContent(content, contentType) with correct contentType and a valid json string as content

This is how I install the features

private val httpClient by lazy {
        HttpClient {
            install(Logging) {
                logger = apiLogger
                level = LogLevel.ALL
            }
            install(JsonFeature) {
                serializer = KotlinxSerializer()
            }
        }
    }

This is how we do the post request inside the same class

    fun fireEvents(event: List<EventRequest>) = blockingGet {
        httpClient.post<Unit>(
            scheme = SCHEME_HTTPS,
            host = baseUrl,
            path = EVENT_API_V1,
            body = event
        ) {
            header(CONTENT_TYPE, APPLICATION_JSON)
            header(API_TOKEN_HEADER, applicationApiToken)
        }
    }

In previous version I used to see the body correctly. But now I see the log just BODY START and BODY END

    METHOD: HttpMethod(value=POST)
    COMMON HEADERS
    -> some-key: some-value
    -> Accept: application/json
    -> Accept-Charset: UTF-8
    CONTENT HEADERS
    BODY Content-Type: application/json
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
I/AppLogger: RESPONSE: 200 OK
    METHOD: HttpMethod(value=POST)
I/AppLogger: FROM: https://something.com/v1/send
    COMMON HEADERS
    -> Connection: keep-alive
    -> Date: Tue, 31 Mar 2020 14:58:04 GMT
    -> Server: nginx/1.14.1
    -> Transfer-Encoding: chunked
    -> X-Android-Received-Millis: 1585666684496
    -> X-Android-Response-Source: NETWORK 200
    -> X-Android-Selected-Protocol: http/1.1
    -> X-Android-Sent-Millis: 1585666684232
I/AppLogger: BODY Content-Type: null
    BODY START
I/AppLogger: BODY END

I am not sure if this is normal or not. But can you tell me if I am doing something wrong in my code that I can fix? I checked changelog but I did not see anything that requires my attention in this code snippet I posted.

Provide the capability to highlight breaking changes on the Changelog page breaking-change

Connect request sends wrong status line

A single slash gets ignored for defining a route, but 1.5 requires them due to KTOR-372

Dispatcher is closing earlier than client

Exception kotlinx.serialization.SerializationException: Class 'ArrayList' is not registered for polymorphic serialization in the scope of 'Collection' in 1.5.0

Prevent double quotes on header params

Release 1.5.1

1.5.0

released 22nd December 2020

Client

Add response text to the message of ResponseException and derived exceptions

Support Java HTTP Client

Ability to send cookies with HttpRequestBuilder

Allow OkHttpConfig to configure WebSocket.Factory

Make curl engine option sslVerify public

CIO: client throws ConnectTimeoutException when endpoint.connectRetryAttempts = 0

Apache client engine sometimes hits an unrecoverable socket timeout when using ChannelWriterContent

https://youtrack.jetbrains.com/issue/KTOR-1149

Test to reproduce below. Tested on two different machines (Windows and macOS), works fine with CIO. With 500 test iterations, happens at least once most times.

import io.ktor.application.*
import io.ktor.client.*
import io.ktor.client.engine.apache.*
import io.ktor.client.features.*
import io.ktor.client.request.*
import io.ktor.http.*
import io.ktor.http.content.*
import io.ktor.request.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.cio.*
import io.ktor.server.engine.*
import io.ktor.utils.io.*
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.AfterEach
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.RepeatedTest
import java.util.concurrent.TimeUnit

class KtorApacheBugTest {

    private val port = 8000
    private val server = embeddedServer(
        factory = CIO,
        port = port,
    ) {
        routing {
            put("/") {
                call.receiveChannel().discard()
                call.respond(HttpStatusCode.OK)
            }
        }
    }
    private val client = HttpClient(Apache) {
        install(HttpTimeout) {
            socketTimeoutMillis = 10_000
            connectTimeoutMillis = 10_000
        }
    }

    @BeforeEach
    fun setUp() {
        server.start()
    }

    @AfterEach
    fun tearDown() {
        client.close()
        server.stop(0, 1, TimeUnit.SECONDS)
    }

    @RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = (0 until 10_000).map { it.toByte() }.toByteArray()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }
}

Another simple test (seems the issue is with ChannelWriterContent) below.

This works:

@RepeatedTest(500)
    fun `does not hit a socket timeout`(): Unit = runBlocking {
        client.put<String>("http://localhost:${port}") {
            body = LocalFileContent(File("gradle/wrapper/gradle-wrapper.jar"))
        }
    }

This does not:

@RepeatedTest(500)
    fun `eventually hits a socket timeout`(): Unit = runBlocking {
        val chunk = File("gradle/wrapper/gradle-wrapper.jar").readBytes()

        client.put<String>("http://localhost:${port}") {
            body = ChannelWriterContent(
                body = {
                    writeFully(chunk)
                },
                contentType = null,
            )
        }
    }

Sending the same via ByteArrayContent works fine as well.

CIO may not work correctly with proxy for https requests

https://youtrack.jetbrains.com/issue/KTOR-1306

Steps:

privoxy setup:

$ cat  ./privoxyconfig 
confdir /etc/privoxy
temporary-directory /tmp
logdir /var/log/privoxy
actionsfile match-all.action # Actions that are applied to all sites and maybe overruled later on.
actionsfile default.action   # Main actions file
actionsfile user.action      # User customizations
filterfile default.filter
filterfile user.filter      # User customizations
logfile privoxy.log
debug     1 # Log the destination for each request Privoxy let through. See also debug 1024.
debug  1024 # Log the destination for requests Privoxy didn't let through, and the reason why.
debug  4096 # Startup banner and warnings
debug  8192 # Non-fatal errors
listen-address  0.0.0.0:8118
toggle  1
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 1
buffer-limit 4096
enable-proxy-authentication-forwarding 0
trusted-cgi-referer http://www.example.org/
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
tolerate-pipelining 1
socket-timeout 300

$ cat ./Dockerfile 
FROM alpine
RUN apk add --no-cache privoxy=3.0.28-r0
COPY privoxyconfig /etc/privoxy/config
EXPOSE 8118
CMD privoxy --no-daemon /etc/privoxy/config

$ docker build . -t privoxy
$ docker run --rm -p 127.0.0.1:8118:8118 privoxy

set proxy as described in https://ktor.io/docs/proxy.html with

fun main(args: Array<String>) {
    val client = HttpClient(CIO) {
        install(UserAgent) {
            agent = "Space Unfurl Bot"
        }

        install(ContentEncoding) {
            gzip()
        }

        engine {
            proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("127.0.0.1", 8118))
        }
    }
    val r = runBlocking(CoroutineName("Get Metadata")) {
        client.get<HttpStatement>("https://example.com").execute { call ->
            "response code: ${call.status.description}"

        }
    }
    println(r)
}

Request: [X] https://example.com CIO hangs for a long time and returns java.lang.IllegalArgumentException: Invalid TLS record type code: 72 [X] http://ya.ru same result (ya.ru redirects to https) [V] http://example.com returns page content [V] Replace CIO with Apache - any requests work perfectly fine.

Client sometimes doesn't do response validation

Core

Staging new public API

https://youtrack.jetbrains.com/issue/KTOR-1195

Problem

Sometimes, we introduce new API but, due to our compatibility guarantees, we can't make it public and forced to introduce it as internal, sometimes with magic to invoke such internals from other modules. Usually, it is intended to make them public at some point in the future, however, because of long minor/major release cycles, it's easy to forget to promote such APIs.

Proposed solution

The solution is to provide an annotation with the following signature

/**
 * API marked with this annotation is intended to become public in the future [version].
 * Usually it means that the API can't be public at the moment of development due to
 * compatibility guarantees restrictions.
 *
 * Marking a public declaration with this annotation makes no sense
 * except for the case when it is also marked with [InternalAPI].

 * Please note that the specified [version] and the fact of making something a candidate is not a guarantee, 
 * so the target version could be changed without any notice or even the promotion could be cancelled at all.
 * 
 * @property version in which the API is planned to be promoted
 */
@InternalAPI // so users shouldn't use this annotation, it's for ktor only
@Retention(AnnotationRetention.SOURCE)
@Target(
    AnnotationTarget.CLASS,
    AnnotationTarget.ANNOTATION_CLASS,
    AnnotationTarget.CONSTRUCTOR,
    AnnotationTarget.FUNCTION,
    AnnotationTarget.PROPERTY,
    AnnotationTarget.PROPERTY_SETTER,
    AnnotationTarget.TYPEALIAS
)
public annotation class PublicAPICandidate(val version: String)

Using this annotation, one could mark some internal API, specifying the desired future version. It is recommended to document such API in spite of that it is internal yet.

When a minor/major version starts (or in the middle of development period), one should search for all annotation usages and collect/notify developers who wanted to make something public. At the end of the version development, we should ensure that there are no annotations with this version remaining.

Every promoted API should be properly documented and have adequate tests.

Example

In 1.4.1 we introduce a property

// In 1.4.1
public class SomeConfig {
    /** Initial documentation */
    @PublicAPICandidate("1.5.0") // the next minor version
    internal var usefulThing: String = ""
}

So no breaking changes, just the internal property.

When we develop 1.5.0 we search for all annotation usages and promote it removing annotation.

// In 1.5.0
public class SomeConfig {
    /**
     * Documentation
     */
    public var usefulProperty: String = ""
}

The other good effect is that by doing this we declare our intention to make something public and declare which version is it. So ktor users may see it, and this avoids unnecessary quesions and improves clarity.

Docs

Documentation for kotlinx.serialization wrong and outdated

Prometheus Registry unavailable for Micrometer

metrics prometheus better documentation

Apache client engine is documented to support HTTP/2 but doesn't

clients-index documentation page incorrect link

Public documentation for Ktor 1.4.0 says, that it's in the development

The kotlinx.serialization feature documentation is using deprecated API

Generator

Fix height of the right panel on the new generator website.

Ktor generator produces erroneous build.gradle.kts if KTS is chosen

Ktor generator fails to generate when Auth-Oauth is selected

DoubleReceive and Kotlinx.serialization features are missing in generator

Infrastructure

settings.gradle: if (native_targets_enabled) does nothing

Server

ktor not closing user connections

https://youtrack.jetbrains.com/issue/KTOR-368

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1885

Ktor Version and Engine Used (client or server and name) 1.3.2, Netty server, routes

ktor_version=1.3.2, kotlin_version=1.3.70

implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version")
    implementation("io.ktor:ktor-server-netty:$ktor_version")
    implementation("ch.qos.logback:logback-classic:$logback_version")
    testImplementation("io.ktor:ktor-server-tests:$ktor_version")

Describe the bug I migrated one of my project form php to ktor and deployed it in production for about 5k users. After one day server stopped working and linux file opened limits exceed which was 20k and when i checked with command ss -ant | grep :8383 | wc -l connections were 20k+ but i don't have users more than 5k which were not all active. Then i switched my floating ip to old php server and no more request to this server next day when i checked again with command ss -ant | grep :8383 | wc -l connections were still 20k+ even no more new request were coming and all connections were in ESTAB state.

So after that i created new project which was empty with ktor, kotlin dsl i mentioned its dependencies above and run that on that server Then i run this command watch -n 1 'ss -ant | grep {MY_IP}' on server to filter connections for my ip first it was no connection from my ip on port 8383 then i send single request from postman on SERVER_IP:8388 which was simple get request and response was from call.respond("Testing") and on server terminal it show me connection on 8383 with my ip. In postman request was complete and i got result. Then i closed postman and keep checking when it will close my connection from server. I checked that after an hour still connection was not closed and when i requested again it was still using that same connection. So it was reusing connection and not closing even after some hours. I'm not sure but may be error with keep alive or http2. I'm not sending any header i request.

I didn't tried any other ktor or kotlin version and I'm not able to reproduce it local. I'm running jar file generated with shadowjar on linux server with systemd service.

To Reproduce

create new project with ktor version 1.3.2 and kotlin 1.3.70 with GradleKotlinDsl

fun main(args: Array<String>): Unit = io.ktor.server.netty.EngineMain.main(args)
@Suppress("unused") // Referenced in application.conf
@kotlin.jvm.JvmOverloads
fun Application.module(testing: Boolean = false) {
    install(Routing) {
        get("/") {
            call.respond("Testing")
        }
    }
}

create jar file with shadowjar

tasks {
    named<ShadowJar>("shadowJar") {
        archiveBaseName.set("shadow")
        mergeServiceFiles()
        manifest {
            attributes(mapOf("Main-Class" to mainClasses))
        }
        transform(com.github.jengelman.gradle.plugins.shadow.transformers.ServiceFileTransformer::class.java) {
            setPath("META-INF/services")
            include("org.eclipse.jetty.http.HttpFieldPreEncoder")
        }
        archiveFileName.set("example.jar")
    }
}

upload on ubuntu server and create its systemd service and run with systemctl start example service file:

[Unit]
Description=example
After=syslog.target

[Service]
User=root
LimitNOFILE=20536
ExecStart=/var/www/example/example-script
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

example-script:

#!/bin/sh
cd /var/www/example
java -jar exmaple.jar

Expected behavior When user make request to http://SERVER_IP:APP_PORT it should close it after request complete but it keep opened it until we will stop server. Checking with ss -ant | grep {CLIENT_PUBLIC_IP}

Screenshots This is screenshot which i get after an hour when made last request in screenshot last connection with port 8383 that is from my machine and it is still not closed. Screenshot 2020-05-20 at 9 32 50 PM

Get client certificate information from request

Support Sealed Classes inside Session-Objects

Implement proper unhandled exception handling strategy

Add option to use specific alias from keystore in CIO TLSConfigBuilder JVM

Support anyHeader() in CORS feature configuration

Serializing collections of different element types

https://youtrack.jetbrains.com/issue/KTOR-1163

Original question from https://stackoverflow.com/questions/64319112/why-serializing-collections-of-different-element-types-is-not-supported-in-ktor

To reproduce use the following code example:

@Serializable
sealed class Message {
    abstract val content: String
}

@Serializable
data class BroadcastMessage(override val content: String) : Message()

@Serializable
data class DirectMessage(override val content: String, val recipient: String) : Message()

val data: List<Message> = listOf(
    DirectMessage("Hey, Joe!", "Joe"),
    BroadcastMessage("Hey, all!")
)

fun main() {
    embeddedServer(Netty, port = 8080, host = "127.0.0.1") {
        install(ContentNegotiation) {
            json()
        }
        routing {
            get("/") {
                call.respond(data)
            }
        }
    }.start(wait = true)
}

Result:

java.lang.IllegalStateException: Serializing collections of different element types is not yet supported. Selected serializers: [DirectMessage, BroadcastMessage]
	at io.ktor.serialization.SerializerLookupKt.elementSerializer(SerializerLookup.kt:71)
	at io.ktor.serialization.SerializerLookupKt.serializerForSending(SerializerLookup.kt:41)
	at io.ktor.serialization.SerializationConverter.convertForSend(SerializationConverter.kt:114)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invokeSuspend(ContentNegotiation.kt:149)
	at io.ktor.features.ContentNegotiation$Feature$install$2.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at ServerKt$main$1$2$1.invokeSuspend(server.kt:39)
	at ServerKt$main$1$2$1.invoke(server.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.routing.Routing.executeResult(Routing.kt:148)
	at io.ktor.routing.Routing.interceptor(Routing.kt:35)
	at io.ktor.routing.Routing$Feature$install$1.invokeSuspend(Routing.kt:100)
	at io.ktor.routing.Routing$Feature$install$1.invoke(Routing.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invokeSuspend(ContentNegotiation.kt:113)
	at io.ktor.features.ContentNegotiation$Feature$install$1.invoke(ContentNegotiation.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invokeSuspend(DefaultEnginePipeline.kt:121)
	at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$2.invoke(DefaultEnginePipeline.kt)
	at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:243)
	at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:113)
	at io.ktor.util.pipeline.SuspendFunctionGun.execute(SuspendFunctionGun.kt:133)
	at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invokeSuspend(NettyApplicationCallHandler.kt:54)
	at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest$1.invoke(NettyApplicationCallHandler.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:55)
	at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
	at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:158)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:56)
	at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
	at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:37)
	at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:29)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Add excludeSuffix to HttpsRedirect feature

Allow any header in CORS configuration

Micrometer metrics: scan of random URLs of a server exhausts available values per tag in influx

Deprecate HTTP/2 push support

Jackson: Crash when sending large responses in 1.4.2

https://youtrack.jetbrains.com/issue/KTOR-1369

Hi there,

We've discovered that Ktor 1.4.2 appears to crash out when sending large responses, using the Jackson feature. I think it might be related to KTOR-1342, but it's also happening on sending. Downgrading to ktor-jackson:1.4.1 seems to fix the issue.

The error logged is:

Exception in thread "eventLoopGroupProxy-3-1" kotlinx.coroutines.CoroutinesInternalError: Fatal exception in coroutines machinery for DispatchedContinuation[UnsafeBlockingTrampoline@2b5e16e4, Continuation at io.ktor.utils.io.jvm.javaio.OutputAdapter$loop$1.loop(Blocking.kt:311)@114e04cb]. Please read KDoc to 'handleFatalException' method and report this incident to maintainers
        at kotlinx.coroutines.DispatchedTask.handleFatalException$kotlinx_coroutines_core(DispatchedTask.kt:93)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:64)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.jvm.javaio.BlockingAdapter$end$1.resumeWith(Blocking.kt:163)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:296)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:184)
        at io.ktor.utils.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:93)
        at io.ktor.utils.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2258)
        at io.ktor.utils.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:929)
        at io.ktor.utils.io.ByteBufferChannel.consumed(ByteBufferChannel.kt:1953)
        at io.ktor.server.netty.cio.NettyResponsePipeline$processBodyFlusher$2.invokeSuspend(NettyResponsePipeline.kt:305)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.ktor.server.netty.EventLoopGroupProxy$Companion$create$factory$1$1.run(NettyApplicationEngine.kt:215)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.ClassCastException: class kotlin.coroutines.jvm.internal.CompletedContinuation cannot be cast to class kotlinx.coroutines.DispatchedContinuation (kotlin.coroutines.jvm.internal.CompletedContinuation and kotlinx.coroutines.DispatchedContinuation are in unnamed module of loader 'app')
        at kotlinx.coroutines.CoroutineDispatcher.releaseInterceptedContinuation(CoroutineDispatcher.kt:103)
        at kotlin.coroutines.jvm.internal.ContinuationImpl.releaseIntercepted(ContinuationImpl.kt:118)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:39)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:55)

I've created a sample project for you to try - https://github.com/SkyeWelch/ktor-1342-error-sample . You can run it with ./gradlew run, and if it's the same as on my machine see the following responses (I'm using httpie to make requests):

→ http localhost:8080/small
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked

{
    "string": "small string"
}

→ http localhost:8080/large
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=UTF-8
transfer-encoding: chunked


http: error: ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Please let me know if you need anything else.

Thanks, Skye

CORS denies simple request where the content type includes a character encoding

GMTDate: Operator overload (plus/minus) for kotlin.time.Duration

Parsing Authorization header throws IllegalArgumentException instead of returning null

Support for precompresed files

Other

Ktor should resolve trailing slash as a different URL

URLBuilder to append paths to currently encoded path

Content-Disposition additional parameters should be inside quotes

Code autoreload not working for me

java.lang.IllegalStateException: Reading is not available in state Reading

https://youtrack.jetbrains.com/issue/KTOR-710

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/813

Ktor Version

1.0.1

Ktor Engine Used(client or server and name)

Netty / Firefox

JVM Version, Operating System and Relevant Context

1.8 Windows Server 2008R2

Feedback

I don't how this happen, I can't reproduce it.

stacktrace:

500 Internal Server Error: POST - /search/TT
java.lang.IllegalStateException: Reading is not available in state Reading
        at kotlinx.coroutines.io.internal.ReadWriteBufferState.startReading$kotlinx_coroutines_io_jvm(ReadWriteBufferSta
te.kt:19)
        at kotlinx.coroutines.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:238)
        at kotlinx.coroutines.io.ByteBufferChannel.access$setupStateForRead(ByteBufferChannel.kt:21)
        at kotlinx.coroutines.io.ByteBufferChannel.lookAhead(ByteBufferChannel.kt:2868)
        at kotlinx.coroutines.io.DelimitedKt.readUntilDelimiter(Delimited.kt:24)
        at io.ktor.http.cio.MultipartKt.copyUntilBoundary(Multipart.kt:339)
        at io.ktor.http.cio.MultipartKt$copyUntilBoundary$1.invokeSuspend(Multipart.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
        at kotlinx.coroutines.DispatchedContinuation.resumeWith(Dispatched.kt:111)
        at kotlinx.coroutines.io.internal.CancellableReusableContinuation.resumeWith(CancellableReusableContinuation.kt:
83)
        at kotlinx.coroutines.io.ByteBufferChannel.resumeWriteOp(ByteBufferChannel.kt:2207)
        at kotlinx.coroutines.io.ByteBufferChannel.bytesRead(ByteBufferChannel.kt:865)
        at kotlinx.coroutines.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:426)
        at kotlinx.coroutines.io.ByteBufferChannel.readAvailable(ByteBufferChannel.kt:559)
        at kotlinx.coroutines.io.ByteBufferChannel.readAvailableSuspend(ByteBufferChannel.kt:593)
        at kotlinx.coroutines.io.ByteBufferChannel$readAvailableSuspend$2.invokeSuspend(ByteBufferChannel.kt)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:32)
        at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:236)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

OAuth feature is incompatible with Dropbox

https://youtrack.jetbrains.com/issue/KTOR-715

This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/771

Ktor Version

1.0.0

Ktor Engine Used(client or server and name)

Server: Netty Client: Apache Using OAuth feature

JVM Version, Operating System and Relevant Context

JVM: 10 OS: Mac

Feedback

OAuth2 with Dropbox fails during the "request access token" step. In summary, the OAuth feature includes the state parameter in the request. Dropbox's API does not allow unexpected parameters, and fails the OAuth access token request.

From what I can tell, there is currently no way to prevent the state from being added to the access token request.

Details

Consider the following 3 client-server exchanges. Here the Client is the Ktor server, and the Server is the identity provider (Dropbox, Google, Github, etc).

[Client] -- /authorize -> [Server]
[Client] <- ...?code=... -- [Server]
[Client] -- /access_token -> [Server]

In a typical OAuth2 exchange, the Client sends a state=<nonce> parameter with the request in 1 and the Server responds with state=<nonce> in the redirect in 2. (This state sending is not mandatory).

The OAuth feature is configured to include the state in 3 when the Client requests the access token from the Server. Some APIs such as Github's want the state to be included in the request. Dropbox however, reject requests that include the state.

The following code sets the state on the access token request. https://github.com/ktorio/ktor/blob/85210082c729b12099ad25f509e99b23ccea23b8/ktor-features/ktor-auth/src/io/ktor/auth/OAuth2.kt#L149-L151 In short, if a state was returned from the Server in 2, it will be non-null at this point in the code.

Using a custom `OAuth2StateProvider`

OAuth2ServerSettings allows the settings of a custom OAuth2StateProvider with which we can generate a custom state. I mentioned previously that the state is not mandatory, so maybe we can not generate a state? This is not an option, as the state must not be null. An empty state results in a request that has state=, but state is still there.