Changelog 2.3 version
2.3.1
released 1st June 2023
Client
AndroidClientEngine: the engine double-parses query parameters before sending a request
We have an API that generates a document served by AWS cloudfront and returns a URL to download the document when it is complete. The problem is that the URL has an extra leading & in the query as in: "?&Expires". When the Ktor client makes the call it drops the leading ampersand which in my mind is correct since it really has no meaning. But unfortunately without the ampersand the call fails.
There is no way to force Ktor not to drop the ampersand.
Cloudfront should not be generating a URL like this and even worse should not be treating the presence as significant and I have a back end person looking into it, but as it stands now we have to have the extra ampersand and will have to hack and expect/actual API to use the platform specific Http Client.
Electron/Node.js detection doesn't work correctly
After testing the KTOR-5650 fix (https://github.com/ktorio/ktor/pull/3456) in Electron the check turned out to return false even though we have the Node API. In Electron at this point there can be two process
objects and the correct one is defined on window.process
, not just process
. So we'd have to first detect which process
object to use and after that the remaining .versions.node
check can be done.
Bearer auth token refresh hangs after prior refresh threw an exception
When BearerAuthProvider
calls AuthTokenHolder.setToken
, it passes a block that is configurable by ktor consumers (unliked BasicAuthProvider
and DigestAuthProvider
). When this block throws an exception, it results in the skipping of newDeferred.complete(newTokens)
and refreshTokensDeferred.value = null
, which causes the next call to setToken
to hang indefinitely on deferred.await()
. Here are the relevant blocks of code:
A workaround for this is for ktor consumers to wrap their refreshTokens
logic in a try block that catches Throwable
like this:
install(Auth) {
bearer {
loadTokens {
// ...
}
refreshTokens {
try {
// ...
} catch (t: Throwable) {
// don't allow this to bubble up
}
}
}
}
Ktor Client Unable to Stream Responses in Javascript
Summary
Ktor 2.3.0 client under Javascript is unable to stream responses from a server. The same code works correctly with Ktor running under the JVM.
Steps to reproduce
- Configure a server to emit responses as a stream
- Configure a ktor client in Javascript to read the response as a stream (e.g.
response.bodyAsChannel()
and reading response line by line)
With the attached sample application
- Run the server ./gradlew :server-app:run and make sure you have a console to look at the server log output
- Launch your browser and open the developer tools
- Visit http://localhost:8080
- Compare the logs from the client and server
Results
Expected
Javascript client reads stream as each item is written
Actual
Javascript buffers until entire response is complete before actually reading it.
Example output from the test application
Server log output:
Writing H at 2023-04-29T16:56:47.240733Z
Writing e at 2023-04-29T16:56:47.750986Z
Writing l at 2023-04-29T16:56:48.256360Z
Writing l at 2023-04-29T16:56:48.761753Z
Writing o at 2023-04-29T16:56:49.264868Z
Writing at 2023-04-29T16:56:49.765610Z
Writing w at 2023-04-29T16:56:50.270883Z
Writing o at 2023-04-29T16:56:50.776197Z
Writing r at 2023-04-29T16:56:51.283105Z
Writing l at 2023-04-29T16:56:51.788978Z
Writing d at 2023-04-29T16:56:52.292238Z
Browser log output:
Received H at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received e at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received l at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received l at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received o at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received w at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received o at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received r at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received l at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Received d at Sat Apr 29 2023 12:56:52 GMT-0400 (Eastern Daylight Time)
Notes
I reproduced this with Ktor 2.3.0. I have not tested older versions to see whether this is a long standing bug or a regression.
A coroutine closed due to cancellation is considered by the JsWebSocketSession to be closed on failure
Expected behavior: Coroutine cancellation leads WebSocket to be closed by Normal Closure (1000) reason or like so
Actual behavior: Any coroutine completion with non-null cause leads WebSocket to be closed by Internal Error (1011) reason
Also, this reason for closing is not supported by browsers, which causes an invalid access error, leaving the WebSocket open.
Server
YAML config does not support reading variables from itself
For example, in HOCON I can write the following:
value {
my = "My value"
}
config {
database {
value = ${value.my}
}
}
If I write similar config file in YAML:
value:
my: "My value"
config:
database:
value: $value.my
It will be failing to read $value.my
assuming it's an environment variable
HOCON: "No configuration setting found for key" error after merging
Steps to reproduce:
- Create a Ktor application
import io.ktor.server.application.*
import io.ktor.server.cio.*
import io.ktor.server.routing.*
fun main(args: Array<String>) {
EngineMain.main(args)
}
@Suppress("unused")
fun Application.main() {
routing {
}
}
- Create two external config files
application.conf
ktor {
development = false
deployment {
host = "0.0.0.0"
port = 8080
shutdownGracePeriod = 2000
shutdownTimeout = 60000
}
application {
modules = [package.MainKt.main]
}
}
extra.conf
// Ktor can't load additional configuration without "ktor" definition
//ktor {
// deployment {
// }
//}
app {
test = "conf"
}
- Run the app using external configuration files
-config=application.conf -config=extra.conf
The result
Actual
It fails with Exception in thread "main" com.typesafe.config.ConfigException$Missing: extra.conf: 1: No configuration setting found for key 'ktor'
Expected
The app should run since the ktor configuration is defined in the application.conf file
As a workaround, it works when empty ktor and deployment configurations are added to the extra.conf file.
The Logging plugin doesn't log full kotlinx deserialization errors
KotlinxSerializationConverter#deserialize()
“hides” actual deserialization error messages:
// KotlinxSerializationConverter.kt
fun deserialize(): Any? {
try {
// Deserialize and return if successful.
} catch (cause: Throwable) {
throw JsonConvertException("Illegal input", cause)
}
}
While the message will be available in the cause
of the thrown JsonConvertException
, it causes the Logging
plugin to display some pretty useless logs:
// Logging.kt
fun logResponseException(log: StringBuilder, request: HttpRequest, cause: Throwable) {
log.append("RESPONSE ${request.url} failed with exception: $cause")
}
The result is logs like this, which unfortunately doesn't provide much value:
RESPONSE https://host/path failed with exception: io.ktor.serialization.JsonConvertException: Illegal input
Is there a way we can provide cause.message
alongside the "Illegal input"
message from KotlinxSerializationConverter
? Perhaps just something like this:
throw JsonConvertException("Illegal input: ${cause.message}", cause)
XForwardedHeaders should set `remoteAddress` in addition to `remoteHost`
Currently, upon receipt of an `X-Forwarded-For` header, the XForwardedHeaders plugin sets the remoteHost
origin property:
If the value extracted from the header is an address rather than a host, Ktor should be setting the remoteAddress
property.
The KDocs for these two properties are:
/** Client address or host name if it can be resolved. ... */
public val remoteHost: String
and
/** Client address. ... */
public val remoteAddress: String
which clearly implies that for an address, these two values should be the same. The kdocs for remoteAddress
even specify that it can be overwridden via the forwarded headers.
Therefore, client code that wants the remote address should clearly be using remoteAddress
, and yet when the XForwardedHeaders plugin is used and the headers resolve to an address, the remoteHost
will contain the resolved remote address, but remoteAddress
will not.
Sessions: Set-Cookie is added on every api request.
Hi, I'm new to Ktor.
please understand if my question is wrong.
when I use session like below
install(Sessions) {
cookie<User>(
USER_COOKIE_NAME,
directorySessionStorage(File(".sessions"), cached = true)
)
I set session on signin
api is called.
and client saved the sessionId and delivered the sessionId to server on getUser
api like below
and server response with Set-Cookie header even getUser
api controller didn't set session.
REQUEST: http://192.168.224.95:8080/getUser
METHOD: HttpMethod(value=POST)
COMMON HEADERS
-> KEY: Header test
-> Accept: application/json
-> Accept-Charset: UTF-8
-> Cookie: user=4879fe5aee4c5b3d;
CONTENT HEADERS
BODY Content-Type: application/json
BODY Content-Type: application/json; charset=UTF-8
BODY START
RESPONSE: 200 OK
METHOD: HttpMethod(value=POST)
FROM: http://192.168.224.95:8080/getUser
COMMON HEADERS
-> Connection: keep-alive
-> Content-Length: 21
-> Content-Type: application/json; charset=UTF-8
-> Set-Cookie: simple-user=4879fe5aee4c5b3d; Max-Age=604800; Expires=Thu, 20 Aug 2020 15:57:08 GMT; Path=/; HttpOnly; $x-enc=URI_ENCODING
-> X-Android-Received-Millis: 1597334141402
-> X-Android-Response-Source: NETWORK 200
-> X-Android-Selected-Protocol: http/1.1
-> X-Android-Sent-Millis: 1597334137063
{"id":"a","name":"b"}
BODY END
so, I checked which part add Set-Session
and found this code on io.ktor.sessions.Sessions
pipeline.intercept(ApplicationCallPipeline.Features) {
val providerData = sessions.providers.associateBy({ it.name }) {
it.receiveSessionData(call)
}
val sessionData = SessionData(sessions, providerData)
call.attributes.put(SessionKey, sessionData)
}
in my past experience, Set-Session
is added only when session is set.
But, my experience is not long, so, kindly understand if I'm wrong.
I would like to ask if this is intended behavior.
Thanks to read.
Please let me know if further information is required.
Significant delay between getting a part and starting reading from its provider for multipart/form-data requests
I have Ktor server in an Android app and I use it to receive files. I would like to display a progress bar in my app for the duration it will take for the receiving file to arrive.
Some sort of a listener with bytesSent/bytesTotal would be perfect
Just to clarify, I am not interested in listening to upload progress from a client, but the time it takes to receive a files from the server.
Right now my incoming files handling code looks like this:
// 1
call.receiveMultipart().forEachPart { part ->
// 2
when (part) {
is PartData.FileItem -> {
val input = part.provider()
// Read from input by chunks
}
else -> {}
}
}
I would like to listen to the progress between (1) and (2). Right now (1) is called, then the file is fully uploaded to the server, and then (2) is called.
Multipart: Support not writing a temporary file for binary data
I am attempting to migrate an existing netty based application to ktor.
One of its features is handling large file uploads (100mb -> 10Gb).
We are handle sensitive personal information and would prefer not write anything to disk.
I can't see a way to implement this using the existing multipart file upload support in Ktor.
Is there a way to directly access a ReadByteChannel for the file and avoid the tmp disk write?
WebSockets: requests to a non-existing route cause server to lock up after responding with 404 (potential DOS)
Platform: JVM
We recently observed our production application servers locking up and becoming unresponsive due to legacy clients making a large number of web socket upgrade requests to a URI for which a previous route
no longer existed. This is equivalent to a denial-of-service attack. We traced the issue back to the simple reproduction below, which also exposes a Ktor web socket client ambiguity on how to handle 404 and other connection issues: the documentation is not clear about what happens in those cases. If catching an exception is the way to go, as in the reproduction below, please update the documentation.
The issue appears to be that the server pipeline for web socket upgrade requests never completes if the client's upgrade request is sent to a URI for which no web socket handler route exists. The attached reproduction starts a server and then attempts to make 1,000 web socket connections. As is, this succeeds and at the end of the test run, roughly 10 coroutines exist on the server.
If the configured route is changed from /test/{id}
to /test
, the server locks up after processing ~200 requests, meaning it doesn't become unresponsive immediately, but only past a certain load. Over 4,000 suspended coroutines exist on the server at the end of the failed test run.
The workaround we're testing right now is to accept and immediately drop the connections to the known legacy route, but that doesn't resolve the issue that a potential attacker who knows that Ktor with web socket support is running on our servers could craft a large number of web socket requests to a different non-existing route and cause application servers to become unresponsive!
Support optional properties in YAML
In HOCON there are 3 scenarios of variable declaration:
a) With default value
myValue = "default value"
myValue = ${?MY_ENV_VAR}
myValue
equals "default value" if one doesn't provide MY_ENV_VAR
b) Without default value and with question mark
myValue = ${?MY_ENV_VAR}
myValue
equals the value of MY_ENV_VAR
or does not exist otherwise (.getProperty
returns null
)
c) Without default value and with no question mark
myValue = $MY_ENV_VAR
myValue
equals the value of MY_ENV_VAR
which must be present (.getProperty
throws an exception in attempt to read MY_ENV_VAR
)
There are only 2 equivalents in YAML:
a) With default value
myValue = "$MY_ENV_VAR:default value"
myValue
equals "default value" if one doesn't provide MY_ENV_VAR
c) With no default value
myValue = $MY_ENV_VAR
myValue
equals the value of MY_ENV_VAR
which must be present (.getProperty
throws an exception in attempt to read MY_ENV_VAR
)
The only way to achieve "b" in YAML is to write ugly code like the following introducing an extra "NULL" string (Or any other token that one believes will never present in real values):
my:
config: "$MY_ENV_VAR:NULL"
And then handling it like the following.
fun ApplicationConfig.yamlStringPropertyOrNull(path: String): String? = propertyOrNull(path)
?.getString()
?.takeIf { it != "NULL" }
fun ApplicationConfig.yamlListProperty(path: String): List<String> = property(path)
.getList()
.filter { it != "NULL" }
val myValue get(): MyConfig = conf.yamlStringPropertyOrNull("my.config")
Test Infrastructure
testApplication: NPE when test server doesn't reply with an HTTP upgrade
When using the example here I get a java.lang.NullPointerException
when running the test. Am I doing something wrong? My hope is to be able to test my WebSocket client of my KMP-app and I got the impression that this might be able to help me?
Using Ktor 2.2.4 and Kotlin 1.8.10.
Here's the stacktrace of the crash. Thanks in advance.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.NullPointerException
at io.ktor.server.testing.TestApplicationEngineJvmKt$handleWebSocketConversationNonBlocking$5.invokeSuspend(TestApplicationEngineJvm.kt:73)
(Coroutine boundary)
at io.ktor.server.testing.TestApplicationEngineJvmKt.handleWebSocketConversationNonBlocking(TestApplicationEngineJvm.kt:68)
at io.ktor.server.testing.client.TestHttpClientEngineBridge.runWebSocketRequest(TestHttpClientEngineBridgeJvm.kt:34)
at io.ktor.server.testing.client.TestHttpClientEngine.execute(TestHttpClientEngine.kt:50)
at io.ktor.client.engine.HttpClientEngine$executeWithinCallContext$2.invokeSuspend(HttpClientEngine.kt:99)
at io.ktor.client.engine.HttpClientEngine$DefaultImpls.executeWithinCallContext(HttpClientEngine.kt:100)
at io.ktor.client.engine.HttpClientEngine$install$1.invokeSuspend(HttpClientEngine.kt:70)
at io.ktor.client.plugins.HttpSend$DefaultSender.execute(HttpSend.kt:138)
at io.ktor.client.plugins.HttpRedirect$Plugin$install$1.invokeSuspend(HttpRedirect.kt:64)
at io.ktor.client.plugins.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:151)
at io.ktor.client.plugins.HttpSend$Plugin$install$1.invokeSuspend(HttpSend.kt:104)
at io.ktor.client.plugins.websocket.WebSockets$Plugin$install$1.invokeSuspend(WebSockets.kt:169)
at io.ktor.client.plugins.HttpCallValidator$Companion$install$1.invokeSuspend(HttpCallValidator.kt:130)
at io.ktor.client.plugins.HttpRequestLifecycle$Plugin$install$1.invokeSuspend(HttpRequestLifecycle.kt:38)
at io.ktor.client.HttpClient.execute$ktor_client_core(HttpClient.kt:191)
at io.ktor.client.statement.HttpStatement.executeUnsafe(HttpStatement.kt:108)
at io.ktor.client.plugins.websocket.BuildersKt.webSocket(builders.kt:241)
at ModuleTest$testConversation$1.invokeSuspend(ModuleTest.kt:15)
at io.ktor.server.testing.TestApplicationKt$testApplication$builder$1$1.invokeSuspend(TestApplication.kt:288)
Caused by: java.lang.NullPointerException
at io.ktor.server.testing.TestApplicationEngineJvmKt$handleWebSocketConversationNonBlocking$5.invokeSuspend(TestApplicationEngineJvm.kt:73)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Other
getTimeMillis has seconds precision on native
Flaky tests in WinHttp engine
Warn when the RateLimit plugin installed after the routing
The Rate limit provider with name RateLimitName (name=KTOR_NO_NAME_RATE_LIMITER) is not configured
or a similar error doesn't explain that the RateLimit
plugin must be installed if it wasn't and that its installation must go before the one of the routing.
Curl sometimes fails with `API function called from within callback`
Darwin engine does not support streaming of request body
Allow access to RateLimiters related to call
RateLimitters for every request key live in memory forever
In case the plugin is set up to have separate limiter per user, this can cause unnecessary memory usage.
Make System Property to Set outgoingToBeProcessed Size for WebSockets
GMTDate timestamp doesn't reflect timezone when created using `Calendar.toDate` method
val calendar = Calendar.getInstance(TimeZone.getTimeZone("Europe/Berlin"))
val timestamp = calendar.toDate(null).timestamp // Equals to GMTDate().timestamp
After creating a new GMTDate
instance from this timestamp, a different date is created.
2.3.0
released 19th April 2023
Client
Prefer Node instead of browser behavior
In the engine there are several places which check first if (PlatformUtils.IS_BROWSER)
but it would be better if those checked for if (PlatformUtils.IS_NODE)
first (i.e. flip the if and else blocks) because for Electron IS_NODE and IS_BROWSER can both be true at the same time, but the browser fetch API is very crippled in comparison to the node-fetch based solution. For example you can't read response headers without explicitly setting up CORS headers in the backend.
In general, instead of having two Booleans (which can both be true) it would be better to have an enum and a single place in the code where the decision of prioritizing Node vs. browser behavior happens. Otherwise such "bugs" of prioritizing crippled browser APIs will happen all over the place and harm Electron integration.
Java engine: Websockets client sends two PONG frames for each PING frame from a server
When the server sends a PING frame, the Ktor client replies with 2 PONG frames.
Maybe the Java web socket client implementation already deals with PING/PONG automatically, and Ktor adds another layer of PING handling, resulting in 2 PONGs, but I haven't checked the code.
It is technically valid to send unsolicited PONG frames as per the web socket specification, but this is not quite what I would expect, and it's breaking Autobahn TestSuite, marking Ktor as failing ping/pong tests.
DarwinClientEngine: a request deadlocks on macOS since 2.2.2
I'm experiencing that the DarwinClientEngine deadlocks on any http requests since 2.2.2 if they are started from the main thread. If I downgrade to 2.2.1 everything works fine.
deadlocks:
HttpClient(
Darwin.create {
pipelining = true
}
)
println("making request..")
val result = httpClient().get("https://www.google.com")
println("status: $result.status")
works:
HttpClient(
Darwin.create {
pipelining = true
}
)
launch(Dispatchers.Default) {
println("making request..")
val result = httpClient().get("https://www.google.com")
println("status: $result.status")
}
Ktor JS websocket client unconfigurable logging
File Websockets.kt contains own LOGGER
internal val LOGGER = KtorSimpleLogger("io.ktor.client.plugins.websocket.WebSockets")
Which can't be configured.
Implementation of KtorSimpleLoggerJs.kt
@Suppress("FunctionName")
public actual fun KtorSimpleLogger(name: String): Logger = object : Logger {
override fun error(message: String) {
console.error(message)
}
override fun error(message: String, cause: Throwable) {
console.error("$message, cause: $cause")
}
override fun warn(message: String) {
console.warn(message)
}
override fun warn(message: String, cause: Throwable) {
console.warn("$message, cause: $cause")
}
override fun info(message: String) {
console.info(message)
}
override fun info(message: String, cause: Throwable) {
console.info("$message, cause: $cause")
}
override fun debug(message: String) {
console.info("DEBUG: $message")
}
override fun debug(message: String, cause: Throwable) {
console.info("DEBUG: $message, cause: $cause")
}
override fun trace(message: String) {
console.info("TRACE: $message")
}
override fun trace(message: String, cause: Throwable) {
console.info("TRACE: $message, cause: $cause")
}
}
It floods logs with unnecessary and useless messages (why not console.trace
?):
TRACE: Sending WebSocket request [object Object]
This logger should be configurable with Logging plugin.
BearerAuthProvider: Token is being refreshed multiple times when queued call finishes with 401 after refresh token succeeds
We are using BearerAuthProvider and experiencing unexpected consecutive refresh token calls in this particular scenario.
Here the sequence of the calls with url, response code, response time and token. The calls are in response order.
- /url1 -> 401 - response time: 310 ms - uses token1
- /url2 -> 401 - response time: 276 ms - uses token1
- /url3 -> 401 - response time: 322 ms - uses token1
- /refresh -> 200 - response time: 390 ms - refresh token 1 to token 2
- /url1 -> 200 - response time: 168 ms - uses token 2
- /url2 -> 200 - response time: 179 ms - uses token 2
- /url3 -> 200 - response time: 189 ms - uses token 2
- /url4 -> 401 - response time: 264 ms - uses token 1
- /refresh -> 200 - response time: 168 ms - refresh token 2 to token 3
...
What we see is that:
- url1, url2, url3 are queued for retry and retried after step 4 in which the refresh token is done. These retries succeed.
- url4 is started when the refresh token is in process but the 401 is received after the refresh token finishes. The 401 failure of url4 leads to another refresh token. We were expecting that this call was going to be retried immediately without a refresh token, just using token2 (refreshed token received at step 4).
The situation gets even worse if we amplify this behaviour to a higher number of calls: e.g. this could lead to url4 failing twice with 401 and this call will be lost and never complete.
DigestAuthProvider: realm sent from the server doesn't participate in the computation of `response`
realm should be set earlier in the function
should be moved to before line 154. This fixes the issues for me.
Missing cause for WebSocketException in JavaHttpWebSocket
Currently in JavaHttpWebSocket.onError
, Ktor throws a WebSocketException
with just the message from the cause:
override fun onError(webSocket: WebSocket, error: Throwable) {
val cause = WebSocketException("${error.message}")
_incoming.close(cause)
_outgoing.cancel()
socketJob.complete()
}
This is not a nice solution in general, because exception messages don't completely inform about the problem. It's the combination of exception type + message that gives the full insight. For instance, with FileNotFoundException: /tmp/foo
, we don't want to just bubble up the message /tmp/foo
, it doesn't really explain the problem.
As a more realistic example, I'm getting a "null"
message from Ktor because some exception in the Java web socket has no message, which makes this stacktrace pretty pointless:
Caused by: org.hildan.krossbow.websocket.WebSocketException: error in Ktor's websocket: null
at org.hildan.krossbow.websocket.ktor.KtorWebSocketConnectionAdapter$incomingFrames$3.invokeSuspend(KtorWebSocketClient.kt:60)
•••
at org.hildan.krossbow.websocket.test.TestUtilsJvmKt.runSuspendingTest(TestUtilsJvm.kt:8)
at org.hildan.krossbow.websocket.test.TestUtilsJvmKt.runSuspendingTest$default(TestUtilsJvm.kt:8)
at org.hildan.krossbow.websocket.test.autobahn.AutobahnClientTestSuite.runAutobahnTestCase(AutobahnClientTestSuite.kt:200)
at org.hildan.krossbow.websocket.test.autobahn.AutobahnClientTestSuite.autobahn_1_1_7_echo_text_payload(AutobahnClientTestSuite.kt:54)
•••
Caused by: io.ktor.client.plugins.websocket.WebSocketException: null
at (Coroutine boundary.()
•••
Caused by: io.ktor.client.plugins.websocket.WebSocketException: null
at io.ktor.client.engine.java.JavaHttpWebSocket.onError(JavaHttpWebSocket.kt:186)
And the chain of causes stops there. See this build scan for a full example stack trace.
Could you please add the cause here to the WebSocketException
?
Add `append(String, List<String>)` overload to `FormBuilder`
I've stumbled upon improper code generated from swagger spec, roughly equivalent to:
append("key[]", listOf("value0", "value1", "etc"))
Although it is easy to convert to something like:
listOf("value0", "value1", "etc").forEach { append("key[]", it) }
Seems presence of catch-all append<T>
is confusing users:
/**
* Appends a pair [key]:[value] with optional [headers].
*/
@InternalAPI
public fun <T : Any> append(key: String, value: T, headers: Headers = Headers.Empty) {
parts += FormPart(key, value, headers)
}
I propose adding additional append
overload to support List<String>
and optionally List<Number>
parameters:
/**
* Appends a pair [key]:[values] with optional [headers].
*/
public fun append(key: String, values: List<String>, headers: Headers = Headers.Empty) {
require(key.endsWith("[]")) {
"Array parameter must be suffixed with square brackets ie `key[]`"
}
values.forEach { value ->
parts += FormPart(key, value, headers)
}
}
Support for CURLOPT_CAINFO and CURLOPT_CAPATH in ktor-client-curl
We are building a Kotlin Native project for desktop. We decided to use ktor-client-curl, as it supports both Mac and Windows. On the other hand, currently the client fails on Windows with the following exception when using any HTTPS URLs:
io.ktor.client.engine.curl.CurlIllegalStateException: TLS verification failed for request: CurlRequestData(url='https://google.com', method='GET', content: 0 bytes). Reason: SSL peer certificate or SSH remote key was not OK
io.ktor.client.engine.curl.CurlIllegalStateException: TLS verification failed for request: CurlRequestData(url='https://google.com', method='GET', content: 0 bytes). Reason: SSL peer certificate or SSH remote key was not OK
at kotlin.Throwable#<init>(Unknown Source)
at kotlin.Exception#<init>(Unknown Source)
at kotlin.RuntimeException#<init>(Unknown Source)
at kotlin.IllegalStateException#<init>(Unknown Source)
at io.ktor.client.engine.curl.CurlIllegalStateException#<init>(Unknown Source)
at io.ktor.client.engine.curl.internal.CurlMultiApiHandler.collectFailedResponse#internal(Unknown Source)
at io.ktor.client.engine.curl.internal.CurlMultiApiHandler.processCompletedEasyHandle#internal(Unknown Source)
at io.ktor.client.engine.curl.internal.CurlMultiApiHandler.handleCompleted#internal(Unknown Source)
at io.ktor.client.engine.curl.internal.CurlMultiApiHandler#perform(Unknown Source)
at io.ktor.client.engine.curl.CurlProcessor.$runEventLoop$lambda$3COROUTINE$17.invokeSuspend#internal(Unknown Source)
at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(Unknown Source)
at kotlinx.coroutines.DispatchedTask#run(Unknown Source)
at io.ktor.util.MultiWorkerDispatcher.$workerRunLoop$lambda$4COROUTINE$80.invokeSuspend#internal(Unknown Source)
at kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(Unknown Source)
at kotlinx.coroutines.DispatchedTask#run(Unknown Source)
at kotlinx.coroutines.EventLoopImplBase#processNextEvent(Unknown Source)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal(Unknown Source)
at kotlinx.coroutines#runBlocking(Unknown Source)
It looks like it's not using the certificate store of Windows, and for me it also failed to detect the ca-cert.pem file regardless where I copied it (see https://github.com/ktorio/ktor/issues/671#issuecomment-619832512 for suggestions).
It would be great if the CURLOPT_CAINFO
and CURLOPT_CAPATH
options would be configurable for the Ktor Curl client as this way I could directly provide the path to the ca-cert.pem file.
ByteBufferChannel throws exception when HttpCache, ContentEncoding and onDownload are used
Description
After updating from KTOR 2.1.2 to 2.2.2 and adding HttpCache
I started to receive bug reports from my testers regarding a GZIP magic invalid: 12345
error. After some testing I realized that only release builds but not debug builds were affected and traced the issue to the LogLevel
used, which is LogLevel.INFO
for release and LogLevel.ALL
for debug. Also LogLevel.HEADERS
is affected by this issue.
Updating to KTOR 2.2.3 did not bring any improvements.
Setup
- KTOR 2.2.2 or 2.2.3
- OkHttp Engine on Android is broken
- Darwin Engine on iOS works as expected
The following configuration triggers the issue. It happens with and without setting the public/private storage set to my storage implementation. To ensure my storage doesn't cause the issue, I commented it out with the same result. Setting one or the other or both makes no difference.
To fix the problem, one of three things needs to happen:
- Remove the
install(HttpCache)
, commenting out the storages as shown does not help - Remove the
install(ContentEncoding)
, maybe removing GZIP is sufficient, but the server doesn't provide Deflate so I can't test this - Set
level
toLogLevel.ALL
install(HttpCache) {
// publicStorage(storage)
// privateStorage(storage)
}
install(ContentEncoding) {
deflate(1.0f)
gzip(0.9f)
}
install(Logging) {
level = LogLevel.INFO
logger = object : Logger {
override fun log(message: String) {
println(message)
}
}
}
Stack trace
java.lang.IllegalStateException: GZIP magic invalid: 958
at java.lang.reflect.Constructor.newInstance0(Native Method)
at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$1.invoke(ExceptionUtilsJvm.kt:3)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$1.invoke(ExceptionUtilsJvm.kt:1)
at io.ktor.utils.io.ExceptionUtilsJvmKt.tryCopyException(ExceptionUtilsJvm.kt:77)
at io.ktor.utils.io.ByteBufferChannelKt.rethrowClosed(ByteBufferChannel.kt:1)
at io.ktor.utils.io.ByteBufferChannelKt.access$rethrowClosed(ByteBufferChannel.kt:1)
at io.ktor.utils.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:39)
at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:34)
at io.ktor.utils.io.ByteBufferChannel.readAvailable$suspendImpl(ByteBufferChannel.kt:1)
at io.ktor.utils.io.ByteBufferChannel.readAvailable(ByteBufferChannel.kt:3)
at io.ktor.utils.io.jvm.javaio.InputAdapter$loop$1.loop(Blocking.kt:126)
at io.ktor.utils.io.jvm.javaio.InputAdapter$loop$1$loop$1.invokeSuspend(Blocking.kt:12)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:12)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:119)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:11)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:28)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter.submitAndAwait(Blocking.kt:16)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter.submitAndAwait(Blocking.kt:3)
at io.ktor.utils.io.jvm.javaio.InputAdapter.read(Blocking.kt:6)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:248)
at java.io.BufferedInputStream.read(BufferedInputStream.java:267)
// my app stack
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:12)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:119)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:13)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:3)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:1)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:15)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:29)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:1)
Caused by: java.lang.IllegalStateException: GZIP magic invalid: 958
at io.ktor.util.EncodersJvmKt$inflate$1.invokeSuspend(EncodersJvm.kt:666)
at io.ktor.util.EncodersJvmKt$inflate$1.invoke(EncodersJvm.kt:1)
at io.ktor.util.EncodersJvmKt$inflate$1.invoke(EncodersJvm.kt:2)
at io.ktor.utis.io.CoroutinesKt$launchChannel$job$1.invokeSuspend(Coroutines.kt:68)
... 8 more
While playing around with logger settings I also got this error. It seems like the file I download determines which of the two exceptions I get. Downloading the same file multiple times consistently reproduces the same exception, across app restarts.
java.lang.IllegalArgumentException: max shouldn't be negative: -22207
at java.lang.reflect.Constructor.newInstance0(Native Method)
at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$1.invoke(ExceptionUtilsJvm.kt:103)
at io.ktor.utils.io.ExceptionUtilsJvmKt$createConstructor$$inlined$safeCtor$1.invoke(ExceptionUtilsJvm.kt:90)
at io.ktor.utils.io.ExceptionUtilsJvmKt.tryCopyException(ExceptionUtilsJvm.kt:66)
at io.ktor.utils.io.ByteBufferChannelKt.rethrowClosed(ByteBufferChannel.kt:2400)
at io.ktor.utils.io.ByteBufferChannelKt.access$rethrowClosed(ByteBufferChannel.kt:1)
at io.ktor.utils.io.ByteBufferChannel.setupStateForRead(ByteBufferChannel.kt:295)
at io.ktor.utils.io.ByteBufferChannel.readAsMuchAsPossible(ByteBufferChannel.kt:2459)
at io.ktor.utils.io.ByteBufferChannel.readAvailable$suspendImpl(ByteBufferChannel.kt:672)
at io.ktor.utils.io.ByteBufferChannel.readAvailable(Unknown Source:0)
at io.ktor.utils.io.jvm.javaio.InputAdapter$loop$1.loop(Blocking.kt:38)
at io.ktor.utils.io.jvm.javaio.InputAdapter$loop$1$loop$1.invokeSuspend(Unknown Source:14)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at io.ktor.utils.io.jvm.javaio.UnsafeBlockingTrampoline.dispatch(Blocking.kt:313)
at kotlinx.coroutines.internal.DispatchedContinuation.resumeWith(DispatchedContinuation.kt:201)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter.submitAndAwait(Blocking.kt:231)
at io.ktor.utils.io.jvm.javaio.BlockingAdapter.submitAndAwait(Blocking.kt:201)
at io.ktor.utils.io.jvm.javaio.InputAdapter.read(Blocking.kt:65)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:248)
at java.io.BufferedInputStream.read(BufferedInputStream.java:267)
// my app stack
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Caused by: java.lang.IllegalArgumentException: max shouldn't be negative: -22207
at io.ktor.utils.io.ByteBufferChannel.discard$suspendImpl(ByteBufferChannel.kt:1667)
at io.ktor.utils.io.ByteBufferChannel.discard(Unknown Source:0)
at io.ktor.util.EncodersJvmKt$inflate$1.invokeSuspend(EncodersJvm.kt:163)
at io.ktor.util.EncodersJvmKt$inflate$1.invoke(Unknown Source:8)
at io.ktor.util.EncodersJvmKt$inflate$1.invoke(Unknown Source:4)
at io.ktor.utils.io.CoroutinesKt$launchChannel$job$1.invokeSuspend(Coroutines.kt:134)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Response of the server
Following file is downloaded
CE3PRO_pikachu_1gen_flowalistik.gcode
{width=70%}
Logging: Add filter/sanitization of sensitive headers
Extracted this comment to a dedicated issue: https://youtrack.jetbrains.com/issue/KTOR-481/Ktor-client-logging-Add-an-option-to-filter-logged-calls#focus=Comments-27-6332247.0-0
I would like to have the ability to log debug headers like request-id trace headers while not logging sensitive Bearer token headers
CIO: Client fails to parse response without Content-Length, Connection headers and chunked transfer encoding
I'm trying to use Ktor to talk to the REST Api of Shelly Smart Home devices, which use an lwIP webserver.
With a simple client set up like follows:
val httpClient = HttpClient(CIO)
any simple GET request fails with IllegalStateException
:
Failed to parse request body: request body length should be specified,
chunked transfer encoding should be used or
keep-alive should be disabled (connection: close)
because the server neither sends Content-Length
, nor Connection
headers. In fact it doesn't send anything but Content-Type
and Server
:
HTTP/1.0 200 OK
Server: lwIP/2.1.2 (http://savannah.nongnu.org/projects/lwip)
Content-Type: application/json
I understand that this service should probably "behave better", but that's out of my control unfortunately. I debugged into io.ktor.http.cio.HttpBody.parseHttpBody
and it seems to be that the issue would be solvable by defaulting to Connection: close
if no Connection
Header is given; and it seems this is (at least according to wikipedia) in HTTP/1.0 what the default should be; only in HTTP/1.1 connections default to being persistent.
Thanks!
Consider quoting `Boolean` during construction of multipart requests
Currently Boolean
is not supported as a value for form data. However it could be treated the same way as Number
is treated.
Core
Native: Read from a closed socket doesn't throw an exception
Ktor 2.1.3
package transport.ktor
import io.ktor.network.selector.*
import io.ktor.network.sockets.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import kotlin.random.*
import kotlin.test.*
class KtorReadingFromClosedSocketBugTest {
@Test
fun `Native - Read from a closed socket doesn't throw an exception`() {
val address = InetSocketAddress("localhost", Random.nextInt(2_000..65_000))
val tcp = aSocket(SelectorManager(EmptyCoroutineContext)).tcp()
val serverSocket = tcp.bind(address)
runBlocking {
launch {
serverSocket.accept()
println("client connected")
}
val clientSocket = tcp.connect(address)
launch {
val clientReadChannel = clientSocket.openReadChannel()
println("clientSocket.isClosed: ${clientSocket.isClosed}")
println("clientReadChannel.isClosedForRead: ${clientReadChannel.isClosedForRead}")
delay(2_000)
println("clientSocket.isClosed: ${clientSocket.isClosed}")
println("clientReadChannel.isClosedForRead: ${clientReadChannel.isClosedForRead}")
try {
println("trying to read from clientReadChannel")
clientReadChannel.readByte()
println("reading from clientReadChannel succeeded^")
} catch (e: Exception) {
println("reading from clientReadChannel failed")
}
}
delay(1_000)
clientSocket.close()
println("clientSocket closed")
}
/*
jvm target (expected behavior) native target (erroneous behavior)
------------------------------ ----------------------------------
client connected client connected
clientSocket.isClosed: false clientSocket.isClosed: false
clientReadChannel.isClosedForRead: false clientReadChannel.isClosedForRead: false
clientSocket closed clientSocket closed
clientSocket.isClosed: true clientSocket.isClosed: false
clientReadChannel.isClosedForRead: true clientReadChannel.isClosedForRead: false
trying to read from clientReadChannel trying to read from clientReadChannel
reading from clientReadChannel failed <reading from clientReadChannel never returns>
*/
}
}
Migrate to the new Kotlin JS IR backend
in 1.9.0 K/JS legacy compiler is deprecated, so Ktor is broken in user projects:
e: Old Kotlin/JS compiler is no longer supported. Please migrate to the new JS IR backend
Update Parameters and Headers DSL to be consistent with stdlib
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1091
Ktor Version
ktor-http
1.1.5 and below
Feedback
Add a parameters { ... }
builder DSL to replace the less discoverable Parameters.build { ... }
companion object function in favor of being more consistent with the stdlib.
sequenceOf(1, 2, 3)
/listOf(1, 2, 3)
-> parametersOf("first", "a")
sequence { ... }
-> parameters { ... }
Docs
Document how to run a Native server app
IntelliJ IDEA Plugin
OpenAPI: `respondRedirect` (permanent and non-permanent) result in unresolved status code
call.respondRedirect("https://www.google.com")
Note: in Ktor, this function is implemented as following:
public suspend fun ApplicationCall.respondRedirect(url: String, permanent: Boolean = false) {
response.headers.append(HttpHeaders.Location, url)
respond(if (permanent) HttpStatusCode.MovedPermanently else HttpStatusCode.Found)
}
Expected: if permanent = false, HttpStatusCode.MovedPermanently
should be the status code, and HttpStatusCode.Found
otherwise.
Actual: no information in OpenAPI file
Ktor modules are not resolved from tests module's application.yaml
While it works fine in main module, if you add a separate YAML config for tests, that's what you'll see:
OpenAPI: Support extensions for unhandled exceptions inside StatusPages
intall(StatusPages) {
exception<E> { ... }
unhandledException()
}
fun StatusPagesConfig.unhandledException() {
call.respond(...)
}
Here, call.respond
from unhandledException
must be present in OpenAPI as well.
OpenAPI: schema type does not always contain reference type if more than one `call.respond` calls is used
Example:
get("/a/b/c") {
call.respond(randomCoordinate())
exampleRespond(Coordinate(10, 20))
}
class Coordinate(val x: Int, val y: Int)
fun randomCoordinate(): Coordinate = Coordinate(Random.nextInt(), Random.nextInt())
fun Route.exampleRespond(value: Any?) {
// A hack to add OpenAPI examples without affection actuall HTTP behaviour:
if (false) {
call.respond(value)
}
}
Expected: OpenAPI shows that /a/b/c
route has #ref/Coordinate
response type reference.
(Ideally, it also has to show value: x: 10 y: 20
as an example value for that response type, but it is not supported yet (KTOR-5683))
Actual: OpenAPI shows that /a/b/c
route has an object
response type.
SOE at io.ktor.ide.oas.KtorOasStatusCodeVisitor.visitCallExpression(KtorOasArgumentVisitors.kt:51) when generation OpenAPI on Qodana Cloud project
To reproduce:
Open Qodana Cloud and try to generate OpenAPI for core-api
module.
Exception:
java.lang.StackOverflowError
at org.jetbrains.kotlin.idea.caches.resolve.ResolutionFacadeWithDebugInfo.analyze(ResolutionFacadeWithDebugInfo.kt:43)
at org.jetbrains.kotlin.idea.caches.resolve.ExtendedResolutionApiKt.analyze(ExtendedResolutionApi.kt:124)
at org.jetbrains.kotlin.idea.caches.resolve.ResolutionUtils.analyze(ResolutionUtils.kt:123)
at org.jetbrains.uast.kotlin.internal.IdeaKotlinUastResolveProviderService.getBindingContext(IdeaKotlinUastResolveProviderService.kt:25)
at org.jetbrains.uast.kotlin.internal.IdeaKotlinUastResolveProviderService.getBindingContextIfAny(IdeaKotlinUastResolveProviderService.kt:28)
at org.jetbrains.uast.kotlin.KotlinInternalUastUtilsKt.analyze(kotlinInternalUastUtils.kt:232)
at org.jetbrains.uast.kotlin.KotlinUastResolveProviderService.getExpressionType(KotlinUastResolveProviderService.kt:309)
at org.jetbrains.uast.kotlin.KotlinUastResolveProviderService.getType(KotlinUastResolveProviderService.kt:304)
at org.jetbrains.uast.kotlin.KotlinUElementWithType.getExpressionType(KotlinUElementWithType.kt:16)
at org.jetbrains.uast.kotlin.KotlinUFunctionCallExpression.getExpressionType(KotlinUFunctionCallExpression.kt:92)
at io.ktor.ide.oas.KtorOasStatusCodeVisitor.visitCallExpression(KtorOasArgumentVisitors.kt:51)
at org.jetbrains.uast.kotlin.KotlinUFunctionCallExpression.accept(KotlinUFunctionCallExpression.kt:165)
at org.jetbrains.uast.kotlin.KotlinUField.accept(KotlinUField.kt:59)
at org.jetbrains.uast.internal.ImplementationUtilsKt.acceptList(implementationUtils.kt:14)
at org.jetbrains.uast.kotlin.AbstractKotlinUClass.accept(AbstractKotlinUClass.kt:99)
at io.ktor.ide.oas.KtorOasStatusCodeVisitor.visitExpression(KtorOasArgumentVisitors.kt:45)
at org.jetbrains.uast.visitor.UastVisitor.visitSimpleNameReferenceExpression(UastVisitor.kt:27)
at org.jetbrains.uast.USimpleNameReferenceExpression.accept(USimpleNameReferenceExpression.kt:19)
at org.jetbrains.uast.UEnumConstant.accept(UVariable.kt:148)
at io.ktor.ide.oas.KtorOasStatusCodeVisitor.visitExpression(KtorOasArgumentVisitors.kt:45)
at org.jetbrains.uast.visitor.UastVisitor.visitSimpleNameReferenceExpression(UastVisitor.kt:27)
at org.jetbrains.uast.kotlin.KotlinUSimpleReferenceExpression.accept(KotlinUSimpleReferenceExpression.kt:40)
at org.jetbrains.uast.internal.ImplementationUtilsKt.acceptList(implementationUtils.kt:14)
at org.jetbrains.uast.kotlin.KotlinUFunctionCallExpression.accept(KotlinUFunctionCallExpression.kt:169)
at org.jetbrains.uast.kotlin.KotlinUField.accept(KotlinUField.kt:59)
at org.jetbrains.uast.internal.ImplementationUtilsKt.acceptList(implementationUtils.kt:14)
at org.jetbrains.uast.kotlin.AbstractKotlinUClass.accept(AbstractKotlinUClass.kt:99)
at io.ktor.ide.oas.KtorOasStatusCodeVisitor.visitExpression(KtorOasArgumentVisitors.kt:45)
...
OpenAPI generation can not be canceled (in case of larger projects it takes some time and cancelation must be possible)
Example: Open Qodana Cloud project and generate OpenAPI for core-api
module.
Computation will run for around 1 minute without an option to cancel.
OpenAPI: support examples for object types
Example:
get("/a/b/c") {
call.respond(Coordinate(10, 20))
}
class Coordinate(val x: Int, val y: Int)
Expected: OpenAPI has to show value: x: 10 y: 20
as an example value for that response type of /a/b/c
request.
Actual: example it is not reflected in OpenAPI.
OpenAPI: Support arrays in example values
Example:
call.respond(MyData(x = 1, y = 2, items = listOf("a", "b", "c")))
Or even:
call.respond(MyData(x = 1, y = 2))
...
class MyData(val x: Int, val y: Int, val items: List<String> = listOf("a", "b", "c"))
OpenAPI should have an example entrance:
examples:
Example#1:
description: ""
value:
x: 1
y: 1
items:
- "a"
- "b"
- "c"
OpenAPI: Analyze examples passed via function calls (support data flow analysis)
Example:
routing {
get("/my/route") {
sendValue(MyObject(1, 2))
}
}
...
fun <T> PipelineContext<Unit, ApplicationCall>.sendValue(value: T) {
call.respond(HttpStatusCode.Accepted, value)
}
Expected: After generating OpenAPI documentation, it should tell about MyObject(1, 2) response with example value based on what was passed to sendValue
function.
OpenAPI: deeply analyze generic types of responses
Example:
data class PaginatedList<T>(
@Expose val count: Long,
@Expose val next: Long?,
@Expose val prev: Long?,
@Expose val items: List<T>,
)
data class ProjectListResponseItem(
@Expose val id: String,
@Expose val name: String,
@Expose val slug: String = name.asSlug,
@Expose val branch: String? = null,
@Expose val commit: String? = null,
@Expose val lastChecked: Instant? = null,
@Expose @NoSlug val lastReportId: String? = null,
@Expose val lastState: ReportState?,
@Expose val url: String? = null,
@Expose val problems: Problems? = null,
@Expose val baselineCount: Int? = null,
)
...
call.respond(results())
...
fun results(): PaginatedList<ProjectListResponseItem>
Expected:
responses:
"200":
description: ""
content:
'*/*':
schema:
$ref: "#/components/schemas/PaginatedList_ProjectListResponseItem"
...
schemas:
PaginatedList_ProjectListResponseItem:
type: "object"
properties:
count:
type: "integer"
format: "int64"
next:
type: "integer"
format: "int64"
prev:
type: "integer"
format: "int64"
items:
type: "array"
items:
$ref: "#/components/schemas/ProjectListResponseItem"
ProjectListResponseItem:
type: "object"
properties:
id:
type: "string"
name:
type: "string"
slug:
type: "string"
branch:
type: "string"
commit:
type: "string"
lastChecked:
$ref: "#/components/schemas/Instant"
lastReportId:
type: "string"
lastState:
type: "string"
enum:
- "UPLOADING"
- "UPLOADED"
- "PROCESSED"
- "FAILED_TO_PROCESS"
- "PINNED"
- "MARKED_FOR_DELETION"
- "OVERWRITTEN"
url:
type: "string"
problems:
$ref: "#/components/schemas/Problems"
baselineCount:
type: "integer"
format: "int32"
OpenAPI: statusCode and other fields from exceptions are not represented in OpenAPI
Example:
There is a StatusPages plugin that handles exceptions:
exception<MyExceptionType> { call, cause ->
val descriptor = ApiDescr(cause.name, cause.message)
call.respond(cause.statusCode, descriptor) // <--- HERE
}
And routes:
get("/first/route") {
throw MyExceptionType1("first name", "Cool message!")
}
get("/second/route") {
throw MyExceptionType2("2nd name", "Bad message :(")
}
And exceptions:
class MyExceptionType(val statusCode: Int, val type: String, val details: String)
class MyExceptionType1:(type: String, details: String): MyExceptionType(HttpStatusCode.BAD_REQUEST, type, details)
class MyExceptionType2:(type: String, details: String): MyExceptionType(HttpStatusCode.UNAUTHORIZED, type, details)
Expected OpenAPI:
/first/route:
get:
description: ""
responses:
"400":
description: "Bad Request"
content:
'*/*':
schema:
$ref: "#/components/schemas/ApiDescr"
examples:
Example#1:
value:
type: "first name"
details: "Cool message!"
/second/route:
get:
description: ""
responses:
"401":
description: "Unauthorized"
content:
'*/*':
schema:
$ref: "#/components/schemas/ApiDescr"
examples:
Example#1:
value:
type: "2nd name"
details: "Bad message :("
components:
schemas:
APIError:
type: "object"
properties:
type:
name: "string"
details:
message: "string"
Actual OpenAPI: nothing is known about response types at all.
Note: even if status codes are known statically (written in exception<...>
handlers), the APIError messages can not be deduced from exceptions.
Modules included from application.conf / application.yaml must be present in OpenAPI
Example:
ktor:
application:
modules:
- a.b.c.myModule1
- a.b.c.myModule2
fun Application.myModule1() {
// routes
}
fun Application.myModule2() {
// status pages
}
Expected: In this case, all exceptions thrown from myModule1
must be handled by StatusPages defined in myModule2
. And this must be represented in OpenAPI.
Actual: OpenAPI generator does not know that myModule2
is also included to the application configuration and hence it does not analyze StatusPages from there because it is not directly referenced.
OpenAPI generator has to check application configuration files to know which other modules it has to analyze.
Endpoints view throws SOE when there are recursive Routing calls
Example:
fun Application.main() {
routing {
f()
}
}
fun Route.f() {
get("/f") {...}
g()
}
fun Route.g() {
route("/g") {
f()
}
}
Expected: at least routes /f
and g/f
are shown.
Actual: no routes are shown, StackOverflow is thrown.
Application method references in YAML are not resolvable if `Application` is imported without wildcard in .kt file
Example:
package a.b.c
import io.ktor.server.Application
fun Application.myModule() { ... }
yaml:
ktor:
application:
modules:
- a.b.c.myModule
Expected: a.b.c.myModule
is a reference in yaml leading to Application.myModule()
.
Actual: it does not resolve.
YAML: Support enviroment variables in YAML config
Example:
- Without default values:
ktor:
deployment:
port: $PORT
- And with default values:
ktor:
deployment:
port: "$PORT:8080"
Expected highlighting:
{width=70%}
Actual: currently, the whole "$PORT:8080" is seen as one single string that is a candidate to be converted to Integer, because variables are not supported.
Search results in wizards doesn't work with spaces / names for plugins should use spaces
Searching for "Content Negotiation" both in the web and IntelliJ wizards gives the following results:
I suppose this is because ContentNegotiation
doesn't contain a space (even though other features do).
I suggest:
- Renaming the feature to
Content Negotiation
- Ignoring whitespace in the fuzzy search.
Support WebSockets endpoints in the Endpoints tool window
Documentation for the Endpoints tool window says that WebSocket endpoints are supported:
The Endpoints tool window provides an aggregated view of both client and server APIs used in your project for HTTP and WebSocket protocols.
It would be useful to have this capability for Ktor projects. Currently, only HTTP routes are displayed.
Server
Support Flow in ktor-serialization
I was surprised that kotlinx.coroutines Flow<T>
is currently not supported by ktor-serialization, Flow is not collected so nothing happens.
What steps will reproduce the issue?
This is a minimal reproducer
fun main(args: Array<String>) {
embeddedServer(Netty, 8080) {
install(ContentNegotiation) {
json()
}
routing {
get("/") {
call.respond(
flow {
emit(1)
delay(1000)
emit(2)
}
)
}
}
}.start(wait = true)
}
What is expected
As coroutines are heavilly used and supported in Ktor, I think ktor-serialization should support kotlinx.coroutines Flow<T>
natively, as it is the official asynchronous Stream on Kotlin coroutines world.
I don't think that the Flow<T>
support should require an additional Plugin, but you can of course disagre with my opinion ;)
The minimal support (on multiplatform) can be to just collect the Flow<T>
as a List<T>
, and then serialize it.
Sequence<T>
should be available for deserialization too.
Bonus 1:
On JVM (with kotlinx.serialization, Jackson and Gson) normal and Flow<T>
serialization can be optimized, and asynchronously write in the OutpuStream (server mode) and read from InpuStream (client mode), or using java.io.Writer
/java.io.Reader
. Examples are available here in a kotlinx.serialization JsonLazySequenceTest.kt
I made a custom implementation to support Flow in Ktor server mode in this class on my kotysa-ktor-r2dbc-coroutines sample project -> by the way if you have a few minutes to read this sample's very short code, and tell me if I made a mistake thanks a lot ^^
Bonus 2:
Flow<T>
could also be used to asynchronously serialize data as Server Sent Events and in Websocket in ktor-websocket-serialization
.
The unofficial application/x-json-stream
(MediaType.APPLICATION_JSON_STREAM) could be supported aswell, even if not supported by Browsers and non standard yet, this format can be useful for server-to-server JSON communication. See this micronaut topic
Can't visit files of staticRootFolder
Support preCompressed with resources
/**
* Support pre-compressed files in the file system only (not just any classpath resource)
* ...
public fun Route.preCompressed(
It looks like preCompressed only works with files() at the moment, why ?
Support `100 Continue`
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/119
We have continue support in netty however it shouldn't be done like that: we have to respond continue (to allow client sending request body) only when handler tries to access request body (call.request.receive()
) or at least once request handler has started (that is not exactly correct as user's code could respond 404)
Websockets: Connection should fail immediately (1002/Protocol Error) when control frame has a payload with more than 125 octets
To reproduce run the following test:
class WebsocketsTest {
private val server = embeddedServer(CIO, host = "127.0.0.1", port = 0) {
install(WebSockets)
routing {
webSocket("/") {
for (frame in incoming) {
outgoing.send(frame)
}
}
}
}
private val client = HttpClient(io.ktor.client.engine.cio.CIO) {
install(io.ktor.client.plugins.websocket.WebSockets)
}
@Before
fun setup() {
server.start(wait = false)
}
@After
fun dispose() {
server.stop()
}
@Test
fun test() = testSuspend {
val session = createSession()
session.outgoing.send(Frame.Ping(ByteArray(126)))
val frame = session.incoming.receive()
assertIs<Frame.Close>(frame)
assertEquals(1002, frame.readReason()?.code)
}
private suspend fun createSession(): ClientWebSocketSession {
val url = "ws://127.0.0.1:${server.resolvedConnectors().first().port}/"
return client.webSocketRawSession { url(url) }
}
}
Websockets: Connection should be failed immediately, since all data frames after the initial data frame must have opcode 0
To reproduce run the following test:
class WebsocketsTest {
private val server = embeddedServer(CIO, host = "127.0.0.1", port = 0) {
install(WebSockets)
routing {
webSocket("/") {
for (frame in incoming) {
outgoing.send(frame)
}
}
}
}
private val client = HttpClient(io.ktor.client.engine.cio.CIO) {
install(io.ktor.client.plugins.websocket.WebSockets)
}
@Before
fun setup() {
server.start(wait = false)
}
@After
fun dispose() {
server.stop()
}
@Test
fun test() = testSuspend {
val session = createSession()
session.outgoing.send(Frame.Text(false, "hello ".toByteArray()))
session.outgoing.send(Frame.Text(true, "world".toByteArray()))
val frame = session.incoming.receive()
assertIs<Frame.Close>(frame)
}
private suspend fun createSession(): ClientWebSocketSession {
val url = "ws://127.0.0.1:${server.resolvedConnectors().first().port}/"
return client.webSocketRawSession { url(url) }
}
}
Non-standard `Content-Type` headers for static files
When providing files via a static
block, a non-standard content-type is used.
Code:
fun Application.main() {
install(Routing)
static("assets") {
files("public")
}
}
File structure:
\- public
|- some-html-file.html
|- some-image.png
\- some-ttf-font.ttf
Making requests to the following urls returns a response with the expected Content-Type
header:
/assets/some-html-file.html
returnsContent-Type: text/hml
/assets/some-image.png
returnsContent-Type: image/png
However, making a request to /assets/some-ttf-font.ttf
returns an unexpected Content-Type
.
Example response:
HTTP/1.1 200 OK
Date: Thu, 08 Dec 2022 21:23:03 GMT
Server: Ktor/2.0.3
Last-Modified: Mon, 15 Aug 2022 22:09:01 GMT
Accept-Ranges: bytes
Content-Length: 335196
Content-Type: application/x-font-ttf
According to the IANA, the correct MIME type for a .ttf
file is font/ttf
.
Here is a list of file types which return an incorrect Content-Type
:
ttf
- Expected:
font/ttf
- Actual:
application/x-font-ttf
- Expected:
woff2
- Expected:
font/woff2
- Actual:
application/octet-stream
- Expected:
woff
- Expected:
font/woff
- Actual:
application/x-font-woff
- Expected:
otf
- Expected:
font/otf
- Actual:
application/x-font-otf
- Expected:
gz
- Expected:
application/gzip
- Actual:
application/x-compressed
- Expected:
The source of the issues is the Mimes.kt
file in ktor-http
module.
CallLogging: add config to avoid logging static file request
I want to avoid logging static file requests,currently I avoid logging static file requests like this:
install(CallLogging) {
level = Level.INFO
filter { !it.parameters.contains("static-content-path-parameter") }
}
Since the pathParameterName
property of StaticContent.kt
is private, I have to manually copy the String. Maybe we can make the pathParameterName
String property of StaticContent.kt
public ,or add configuration items in the CallLogging plugin.
The '425 Too Early' status code is missing in the HttpStatusCode enum
Feature request: SO_REUSEADDR option for embedded server
My Environment
iosX64 Native - iOS 16.0 Simulator
Kotlin: 1.8.0
XCode: 14.0.1
Ktor Version: 2.2.3
Background
I'm running a Ktor server for mocking purposes on an iOS simulator. The problem is that when killing the application through XCode, I get no callback to stop the server gracefully (i.e. it's a SIGKILL).
If a request was in progress when the app is killed, it leads to the following exception upon restarting the app (unless I wait about 30 seconds).
io.ktor.utils.io.errors.PosixException.AddressAlreadyInUseException: EADDRINUSE
Feature Request
(I'm a mobile engineer so apologies if what I'm asking is incorrect). Can there be a flag added to the embeddedServer
to allow port reuse, so that killing and restarting the app with the same server port does not lead to a crash?
I.e exposing the API used here to the embeddedServer
configuration: https://youtrack.jetbrains.com/issue/KTOR-4442/EADDRINUSE-on-socket-bind-on-Linux.
Minimal Reproducible Example
https://github.com/SamCosta1/ktor-connection-bug-example/tree/ktor-killing-app-example
Notes
I've not experienced this issue running on Android, this seems to be Native only.
Support loading multiple configuration files
We would like to provide a solution for loading multiple configuration files, i.e. java -jar sample-app.jar -config=application.conf -config=applicationFoo.conf -config=applicationBar.conf.
The mechanism should work from left to right - it means if application.conf and applicationFoo.conf have common properties then values from application.conf should be overwritten by values from applicationFoo.conf, like shown below:
application.conf
ktor {
deployment {
port = 8080
}
application {
class = <org.company.ApplicationClass>
}
}
applicationFoo.conf
ktor {
deployment {
port = 8085
testProperty = "TestValue"
}
}
config {
configurationProperty = "Config value"
}
result configuration:
ktor {
deployment {
port = 8085
testProperty = "TestValue"
}
application {
class = <org.company.ApplicationClass>
}
}
config {
configurationProperty = "Config value"
}
"Serializer for class 'Any' is not found" error when responding with Any type since Ktor 2.2.4
This used to work in 2.2.3:
fun main() {
embeddedServer(Netty, port = 8082, host = "0.0.0.0", module = Application::module)
.start(wait = true)
}
fun Application.module() {
install(ContentNegotiation) {
json()
}
routing {
get {
call.respond(value())
}
}
}
// fun value(): Data = Data("bar") works
fun value(): Any = Data("bar")
@Serializable
data class Data(val foo: String)
on 2.2.4 ktor throws
2023-03-21 01:45:13.069 [eventLoopGroupProxy-4-1] ERROR ktor.application - Unhandled: GET - /
kotlinx.serialization.SerializationException: Serializer for class 'Any' is not found.
Mark the class as @Serializable or provide the serializer explicitly.
CIO: nmap crashes server with "SocketException: Invalid argument" error
so I have a simplest ktor app
fun main() {
val env = applicationEngineEnvironment {
module {
routing {
get("/") {
call.respondText("Hello, world!")
}
}
}
connector {
this.port = 8080
}
}
embeddedServer(CIO, env).start(wait = true)
}
implementation("io.ktor:ktor-server-core:2.2.3")
implementation("io.ktor:ktor-server-cio:2.2.3")
if you perform a scan of open tcp ports like this nmap localhost -Pn -p8080
, the app crashes with
Exception in thread "DefaultDispatcher-worker-2" java.net.SocketException: Invalid argument
at java.base/sun.nio.ch.Net.setIntOption0(Native Method)
at java.base/sun.nio.ch.Net.setSocketOption(Net.java:455)
at java.base/sun.nio.ch.Net.setSocketOption(Net.java:393)
at java.base/sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:280)
at io.ktor.network.sockets.ServerSocketImpl.accepted(ServerSocketImpl.kt:51)
at io.ktor.network.sockets.ServerSocketImpl.acceptSuspend(ServerSocketImpl.kt:42)
at io.ktor.network.sockets.ServerSocketImpl.access$acceptSuspend(ServerSocketImpl.kt:12)
at io.ktor.network.sockets.ServerSocketImpl$acceptSuspend$1.invokeSuspend(ServerSocketImpl.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:42)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
Suppressed: kotlinx.coroutines.DiagnosticCoroutineContextException: [LazyStandaloneCoroutine{Cancelling}@23a3bd4f, Dispatchers.IO]
OAuth2: "JsonObject is not a JsonPrimitive" error when server replies with nested JSON object on a token request
Hi,
I am trying to integrate with a service using OAuth2, however the response that the service gives does not get handled correctly by ktor. From the Asana website, this is a typical response given (https://developers.asana.com/docs/oauth)
{
"access_token": "f6ds7fdsa69ags7ag9sd5a",
"expires_in": 3600,
"token_type": "bearer",
"data": {
"id": 4673218951,
"gid": "4673218951",
"name": "Greg Sanchez",
"email": "gsanchez@example.com"
},
"refresh_token": "hjkl325hjkl4325hj4kl32fjds"
}
Currently ktor expects the response to only contain primitives. When trying to get the response after exchanging the code for the token, I get this error message from ktor.
Element class kotlinx.serialization.json.JsonObject is not a JsonPrimitive
I was wondering if this could be supported, possibly by adding these as extra parameters
Thanks!
CallLogging: logs caused by an exception are suppressed when mdc provider is configured
Ktor version: 2.2.4
Kotlin version: 1.8.10
When running ktor server like:
val server = embeddedServer(Netty, port = 28080) {
install(CallLogging) {
callIdMdc()
}
routing {
get("/test") {
this.call.respondText("Hello, world!")
}
get("/err") {
throw Exception("Unexpected")
}
}
}
It will not log calls to /err route:
@Test
fun testRoot() = runTest {
server.start(false)
val client = HttpClient(CIO)
assertEquals(200, client.get("http://localhost:28080/test").status.value)
assertEquals(500, client.get("http://localhost:28080/err").status.value)
client.close()
server.stop()
}
Produced logs:
2023-03-16 10:14:44,668 INFO [] Application started in 0.284 seconds. [ktor.application] [oroutine#2]
2023-03-16 10:14:44,669 INFO [] Application started: io.ktor.server.application.Application@4f3356c0 [ktor.application] [oroutine#2]
2023-03-16 10:14:45,043 INFO [] Responding at http://0.0.0.0:28080 [ktor.application] [oroutine#1]
2023-03-16 10:14:45,641 INFO [] 200 OK: GET - /test in 163ms [ktor.application] [handler#16]
2023-03-16 10:14:45,669 ERROR [] Unhandled: GET - /err [ktor.application] [handler#29]
java.lang.Exception: Unexpected
(Coroutine boundary)
at io.ktor.util.pipeline.PipelineKt$execute$2.invokeSuspend(Pipeline.kt:478)
at io.ktor.server.routing.Routing.executeResult(Routing.kt:188)
2023-03-16 10:14:47,492 INFO [] Application stopping: io.ktor.server.application.Application@4f3356c0 [ktor.application] [oroutine#2]
2023-03-16 10:14:47,493 INFO [] Application stopped: io.ktor.server.application.Application@4f3356c0 [ktor.application] [oroutine#2]
When CallLogin plug is not configured with mds, it will produces logs:
2023-03-16 10:13:47,070 INFO [] Application started in 0.269 seconds. [ktor.application] [oroutine#2]
2023-03-16 10:13:47,071 INFO [] Application started: io.ktor.server.application.Application@6ac45c0c [ktor.application] [oroutine#2]
2023-03-16 10:13:47,432 INFO [] Responding at http://0.0.0.0:28080 [ktor.application] [oroutine#1]
2023-03-16 10:13:48,031 ERROR [] Unhandled: GET - /err [ktor.application] [handler#29]
java.lang.Exception: Unexpected
(Coroutine boundary)
at io.ktor.util.pipeline.PipelineKt$execute$2.invokeSuspend(Pipeline.kt:478)
at io.ktor.server.routing.Routing.executeResult(Routing.kt:188)
2023-03-16 10:13:48,170 INFO [] 200 OK: GET - /test in 312ms [ktor.application] [handler#16]
2023-03-16 10:13:48,717 INFO [] 500 Internal Server Error: GET - /err in 730ms [ktor.application] [handler#29]
2023-03-16 10:13:49,761 INFO [] Application stopping: io.ktor.server.application.Application@6ac45c0c [ktor.application] [oroutine#2]
2023-03-16 10:13:49,762 INFO [] Application stopped: io.ktor.server.application.Application@6ac45c0c [ktor.application] [oroutine#2]
Static files filters or something similar to mod_rewrite
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/233
I want Ktor to drive my website. I have a bunch of HTML files but I want to strip the HTML suffix from those files when they are being served as the HTML suffix is just a technical detail that I want to strip.
Built-in support for HEAD requests for static files
When serving static files in static
routes using the files(dir)
DSL, it would be nice if a HEAD
mapping was setup in addition to the GET
one, in order to respect HTTP semantics.
The HEAD
mapping should return (among others) the correct status code (e.g. 404 if the file doesn't exist) and also the file size as Content-Length
for instance.
I stumbled upon this issue when trying to serve my local .m2 directory as static files to emulate a remote server, and Ivy (and likely Maven and Gradle) makes use of HEAD requests.
Ability to set Content-Type of static resource
I had this simple route, serving the content of the assets
resource folder when getting assets/*
static("assets") {
resources("assets")
}
I realized that Ktor was responding to these requests with a weird Content-Type
for files ending in .map
like *.js.map
(or *.css.map
I guess).
This doesn't cause any real issue as most clients getting this kind of files ignore content type entirely. Still, I wanted to change that content type to something a little more standard like application/octet-stream
or even something less standard but more useful for debugging, application/json
.
So I tried to change this content type and couldn't find a way. I tried with an interceptor but you're somehow not allowed to change the content type header.
I ended up having to drop the use of static
/resources
. So now I have this:
val oneYearInSeconds = Duration.ofDays(365).toSeconds()
val pathParameterName = "path"
get("/assets/{$pathParameterName...}") {
val relativePath = call.parameters
.getAll(pathParameterName)
?.joinToString(File.separator)
?: return@get
val content = call.resolveResource(relativePath, "assets") { extension ->
when {
extension.endsWith(".js.map") -> ContentType.Application.Json
else -> ContentType.defaultForFileExtension(extension)
}
} ?: return@get
call.response.header(HttpHeaders.CacheControl, "public, max-age=$oneYearInSeconds, immutable")
call.respond(content)
}
Which is kinda longer and unfortunately doesn't use what's already there for static resources.
(I also implemented caching there, because 1. there does not seem to be caching enabled by default for static content and 2. Ktor's caching headers feature kinda sucks, but that's a different story)
Metrics: ClassCastException when the DropwizardMetrics plugin is installed after the MicrometerMetrics plugin
Using the code found in the micrometer-metrics code snippet, and adding the dependencies for the dropwizard-metrics, if the plugin installation block for the Dropwizard plugin is added after the MicrometerMetrics plugin:
install(MicrometerMetrics) {
registry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
distributionStatisticConfig = DistributionStatisticConfig.Builder()
.percentilesHistogram(true)
.maximumExpectedValue(Duration.ofSeconds(20).toNanos().toDouble())
.serviceLevelObjectives(
Duration.ofMillis(100).toNanos().toDouble(),
Duration.ofMillis(500).toNanos().toDouble()
)
.build()
meterBinders = listOf(
ClassLoaderMetrics(),
JvmMemoryMetrics(),
JvmGcMetrics(),
ProcessorMetrics(),
JvmThreadMetrics(),
FileDescriptorMetrics(),
UptimeMetrics()
)
}
install(DropwizardMetrics) {
JmxReporter.forRegistry(registry)
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS)
.build()
.start()
}
then when the /metrics
endpoint is scrapped, an exception is thrown;
java.lang.ClassCastException: class io.ktor.server.metrics.dropwizard.CallMeasure cannot be cast to class io.ktor.server.metrics.micrometer.CallMeasure (io.ktor.server.metrics.dropwizard.CallMeasure and io.ktor.server.metrics.micrometer.CallMeasure are in unnamed module of loader 'app')
at io.ktor.server.metrics.micrometer.MicrometerMetricsKt$MicrometerMetrics$2$4.invoke(MicrometerMetrics.kt:148)
at io.ktor.server.metrics.micrometer.MicrometerMetricsKt$MicrometerMetrics$2$4.invoke(MicrometerMetrics.kt:146)
at io.ktor.server.application.hooks.ResponseSent$install$1.invokeSuspend(CommonHooks.kt:110)
at io.ktor.server.application.hooks.ResponseSent$install$1.invoke(CommonHooks.kt)
at io.ktor.server.application.hooks.ResponseSent$install$1.invoke(CommonHooks.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
at io.ktor.util.pipeline.SuspendFunctionGun.proceedWith(SuspendFunctionGun.kt:88)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invokeSuspend(DefaultTransform.kt:29)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)
at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations$1.invoke(DefaultTransform.kt)
at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
at io.ktor.server.engine.DefaultEnginePipelineKt.tryRespondError(DefaultEnginePipeline.kt:122)
at io.ktor.server.engine.DefaultEnginePipelineKt.handleFailure(DefaultEnginePipeline.kt:54)
at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:40)
A workaround appears to be registering the Dropwizard plugin before the MicrometerMetrics plugin
Support serving static files from resources in GraalVM native image
It is now not possible to serve static files from resources in GraalVM native image with default configuration:
routing {
static("/assets") {
resources("assets")
}
}
It is because internal fun resourceClasspathResource
supports only file
, jar
and jrt
protocols.
But in native image, resources are supported by resource
protocol.
The default implementation of challengeFunction is empty, causing no session users to access protected resources
Module: ktor-server-auth-jvm-2.2.2.jar
Class: io/ktor/server/auth/SessionAuth.kt:123
This line of code does not work for unauthorized users:
session<UserSession>("auth-session")
Since the default challengeFunction parameter is an empty implementation, users without a Session can also access resources protected by auth-session
{width=70%}
install(Authentication) {
form("auth-form") {
validate { credentials ->
if (credentials.name == "123" && credentials.password == "123") {
UserIdPrincipal(credentials.name)
} else {
null
}
}
}
session<UserSession>("auth-session") // not work
}
Support regex patterns in routing
Wrong ContentType for .mjs files
Steps to reproduce: serve static JS module files from a folder in a standard way like
routing {
trace { application.log.trace(it.buildText()) }
static("/scripts/") {
this.staticRootFolder = File(rootPath)
files(".")
}
}
The problem: the MIME type for .mjs files is wrong and the browser cannot load the modules.
Error in the browser console:
Loading module from “http://localhost:81/scripts/asdf.mjs” was blocked because of a disallowed MIME type (“application/octet-stream”).
Expected result: ContentType header should be "text/javascript"
Allow passing multiple acceptable content types to accept route selector
This issue was imported from GitHub issue: https://github.com/ktorio/ktor/issues/1781
Subsystem
Server, route building
Is your feature request related to a problem? Please describe.
Right now the accept()
route selector only takes a single content type.
Describe the solution you'd like
It would be convenient if we could pass more than one, and it could match on any of the ones provided.
Motivation to include to ktor
That would reduce the amount of duplication when multiple content types are handled by the same handler.
Conflict between `ContentNegotiation` and `Mustache` plugins
If I install(ContentNegotiation) { gson() }
and install(Mustache)
call.respond(MustacheContent("temp.hbs", mapOf("user" to sampleUser)))
stops working correctly
Instead of returning proper html output, the server will return a json
{"template":"temp.hbs","model":{"user":{"id":1,"name":"John"}},"contentType":{"contentType":"text","contentSubtype":"html","content":"text/html","parameters":[{"name":"charset","value":"UTF-8","escapeValue":false}]}}
the return content type is application/json
After removing install(ContentNegotiation) { gson() }
block, html is responded correctly but other api routes throw serialisation related errors as expected due to lack of gson
Websockets: connection should be failed immediately when no continuation frame goes after a fragmented text frame
To reproduce send a text Message fragmented into 2 fragments, then Continuation Frame with FIN = false where there is nothing to continue, then an unfragmented Text Message, all sent in one chop.
Expected behavior: the connection is failed immediately since there is no message to continue.
Actual behavior: two text frames are sent back.
embeddedServer(CIO, host = "127.0.0.1", port = 0) {
install(WebSockets)
routing {
webSocket("/") {
for (frame in incoming) {
outgoing.send(frame)
}
}
}
}
Shared
Websockets: Erroneous trace log about expired websocket pings
I have written a simple websocket server with the following configuration:
install(WebSockets) {
pingPeriod = Duration.ofSeconds(30)
}
Here is an excerpt of the logs:
2023-03-19 17:49:25.020 [DefaultDispatcher-worker-25] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: sending ping frame
2023-03-19 17:49:25.023 [DefaultDispatcher-worker-39] TRACE io.ktor.websocket.WebSocket - WebSocketSession(StandaloneCoroutine{Active}@5a55a4c1) receiving frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:49:25.023 [DefaultDispatcher-worker-39] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: received valid pong frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:49:25.023 [DefaultDispatcher-worker-39] TRACE io.ktor.websocket.WebSocket - WebSocket pinger has timed out
2023-03-19 17:49:55.028 [DefaultDispatcher-worker-5] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: sending ping frame
2023-03-19 17:49:55.029 [DefaultDispatcher-worker-52] TRACE io.ktor.websocket.WebSocket - WebSocketSession(StandaloneCoroutine{Active}@5a55a4c1) receiving frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:49:55.029 [DefaultDispatcher-worker-18] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: received valid pong frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:49:55.029 [DefaultDispatcher-worker-18] TRACE io.ktor.websocket.WebSocket - WebSocket pinger has timed out
2023-03-19 17:50:25.035 [DefaultDispatcher-worker-30] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: sending ping frame
2023-03-19 17:50:25.037 [DefaultDispatcher-worker-15] TRACE io.ktor.websocket.WebSocket - WebSocketSession(StandaloneCoroutine{Active}@5a55a4c1) receiving frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:50:25.038 [DefaultDispatcher-worker-15] TRACE io.ktor.websocket.WebSocket - WebSocket Pinger: received valid pong frame Frame PONG (fin=true, buffer len = 76)
2023-03-19 17:50:25.038 [DefaultDispatcher-worker-15] TRACE io.ktor.websocket.WebSocket - WebSocket pinger has timed out
You can see that each time the server sends a PING
frame, it immediately receives a PONG
one, but each time we always get the following log:
TRACE io.ktor.websocket.WebSocket - WebSocket pinger has timed out
This is NOT true, the pong frame has been received in time (which is the default here, 15 seconds).
LOGGER.trace("WebSocket pinger has timed out")
if (rc == null) {
// timeout
// we were unable to send the ping or hadn't got a valid pong message in time,
// so we are triggering close sequence (if already started then the following close frame could be ignored)
onTimeout(CloseReason(CloseReason.Codes.INTERNAL_ERROR, "Ping timeout"))
break
}
Shouldn't we put the log in the condition body? Currently, this is confusing.
Test Infrastructure
runBlocking in TestApplicationEngine loses coroutineContext
TestApplicationEngine
has a few runBlocking
calls inside it.
This can cause trouble for advanced testing, when a suspend function is tested with a coroutineContext
.
One notable example is Kotest. In order to use withClue
, coroutineContext
should contain errorCollectorContextElement
. Otherwise context switching might result in exception. Right now there is no way to pass it into internal runBlocking
calls of TestApplicationEngine
, and it isn't passed from parent context.
Here's a sample code which reproduces the situation above:
@Test
fun test() = runBlocking(errorCollectorContextElement) {
testApplication {
withContext(Dispatchers.Unconfined) {
withClue("some hint") {
withContext(Dispatchers.Unconfined) {
delay(2)
1 shouldBe 1
}
}
}
}
}
Kotest throws an IndexOutOfBoundsException
here, because an error hint is popped from wrong errorCollector (one from another thread).
This particular error can be fixed on Kotest side, but I assume where might be more situations similar to this. Another example might be Exposed framework. One can create a wrapper transaction to rollback it after the test is finished, and context switching might result in losing context with the wrapper transaction.
Other
Upgrade Client Apache Engine Version to use Apache 5
Simplify Static Content Plugin
Design defaults for the static content plugin to make it work out of the box.
Update reported dependencies
Please use separate commit for every dependency
Requests don't match in nested Regex Routing
routing {
route(Regex("""/abc/""")) {
get(Regex("""cde""")) {
call.respondText { "OK" }
}
}
}
None of the /abc/cde
and /abc//cde
is matched
IllegalArgumentException in Regex Routing
routing {
get(Regex("""\(?<a>\)""")) {
call.respondText { "OK" }
}
}
On request GET /<a>)
there is IllegalArgumentException: No group with name <a>
Unneeded escaping in Regex Routing isn't processed
get(Regex("""\/abc"""))
should have the same behavior as get(Regex("""/abc"""))
Reading response of HEAD request breaks when Content-Length > 0
Add opportunity to pass type info into WebSockets serializing methods
Update JTE to 2.3.0
Update Kotlin to 1.8.10
AutoHead should dispose response body
Comparable HttpStatusCode
Subsystem
Client/Server
Motivation
It is much more useful to compare status codes between each other. For example:
httpResponse.status < HttpStatusCode.OK
Currently, in case, if I want to compare status codes I should use next comparison:
httpResponse.status.value < HttpStatusCode.OK.value
Solution
In my PR HttpStatusCode
will implements Comparable
interface and realize compareTo
with any other HttpStatusCode
Add shutdown configuration for engine in stop method
There are two parameters that are used for shutdown configuration: gracePeriod
and shutdownTimeout
`. There are used in ShutdownHook and stop method
In KTOR-5359 configuration for ShutdownHook
was added
Incorrect handling of private cache directive in HttpCachePlugin
Multiple problems related to private responses caching:
- private cache responses can be shared between multiple users when using HttpClient in a proxy use case
s-maxage
had a typo and was used for private responses, while should be used for shared- only first
Cache-Control
header was checked, but they should be merged