Skip to content

Memory problem ? #86

Description

@Haraade

I have had memory problems in dnscrypt server-docker images.
Memory usage is growing rapidly.
Come across this:
tikv/rust-prometheus#424
Could prometheus v0.13.0 be the problem?
Compiled encrypted-dns with prometheus v0.12.0 and it looks like memory problem is gone.
This was done on a singleboard computer arm32v7 with 2Gbyte of LPDDR3 RAM.

Activity

  1. Haraade commented on May 21, 2022

    @Haraade
    Author

    Do not know what does this, tried on another arm32v7 board. CPU (1-core of 4) usage also goes up to 100%

    dnscrypt-server

  2. jedisct1 commented on May 21, 2022

    @jedisct1
    Member

    Can you double check that the issue goes away when downgrading the prometheus crate to v0.12.0?

    How did you do that?

  3. Haraade commented on May 21, 2022

    @Haraade
    Author

    The weird thing is that one device works with prometheus v0.12.0 and another does not (release 0.9.1), but both devices work well with encrypted-dns release 0.3.23 and arm32v7/ubuntu:22.04

    Build dnscrypt-server-docker, FROM arm32v7/ubuntu:22.04

  4. Haraade commented on May 21, 2022

    @Haraade
    Author

    I have also tried prometheus v0.13.1, but there are also issues.
    tikv/rust-prometheus#442

  5. jedisct1 commented on May 21, 2022

    @jedisct1
    Member

    Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

  6. Haraade commented on May 21, 2022

    @Haraade
    Author

    Does it still happen if you don't enable the metrics feature? (disabling it removes prometheus)

    I have to try.

  7. Haraade commented on May 22, 2022

    @Haraade
    Author

    disabled metrics in encrypted-dns.toml.in and -M option when initializing the docker image.
    Memory and CPU usage are growing, and are running in loop.

    dnscrypt-server
    dnscrypt-server

  8. jedisct1 commented on May 22, 2022

    @jedisct1
    Member

    Ok, so at least we know that this is unrelated to Prometheus.

    In case this is not in a dependency, would you be able to pinpoint a specific commit that introduced this?

  9. Haraade commented on May 22, 2022

    @Haraade
    Author

    Version 0.3.23 works.
    cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev 631ebd4

    Version 0.9.0 does not work.
    cargo install --git https://github.com/DNSCrypt/encrypted-dns-server --rev a178d25

    So there must be something between these versions.

    Will probably have to test more to find the specific commit that introduced this.

  10. Haraade commented on May 24, 2022

    @Haraade
    Author

    I think this problem is due to jemallocator.

    See what was mentioned earlier.
    jemalloc/jemalloc#1497
    gnzlbg/jemallocator#168
    rust-lang/rust#83152

    I have tried with tikv-jemallocator. Compiled fine and without memory problem, but unfortunately no response from encrypted-dns. Only gets TIMEOUT on server in dnscrypt proxy.
    tikv-jemallocator supports arm32v7 ?

  11. jedisct1 commented on May 24, 2022

    @jedisct1
    Member

    So, like way too many Rust crates, jemalloc is abandoned, and the maintained fork doesn't work on ARM? Great :(

    As an experiment, 748b944 replaces jemalloc with mimalloc. Does in improve things on your end?

  12. jedisct1 commented on May 24, 2022

    @jedisct1
    Member

    The app doesn't seem to compile at all on armv7 :(

    error[E0308]: mismatched types
       --> /home/pi/.cargo/registry/src/github.com-1285ae84e5963aae/prometheus-0.13.1/src/process_collector.rs:192:13
        |
    190 |     static ref CLK_TCK: i64 = {
        |                         --- expected `i64` because of return type
    191 |         unsafe {
    192 |             libc::sysconf(libc::_SC_CLK_TCK)
        |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `i64`, found `i32`
    
  13. Haraade commented on May 24, 2022

    @Haraade
    Author

    tikv/rust-prometheus#442
    I use prometheus v0.12.0
    Trying mimalloc.

  14. jedisct1 commented on May 24, 2022

    @jedisct1
    Member

    After removing prometheus' process feature, it works for me on armv7l (raspberry Pi with 32-bit raspbian).

  15. Haraade commented on May 25, 2022

    @Haraade
    Author

    Update.
    Has compiled with the latest version of tikv-jemallocator (0.5.0), which was released today and server works excellent.
    tikv/jemallocator#26
    This issue has been resolved. Thank you so much

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions