AIX Open Source

AIX Open Source

Share your experiences and connect with fellow developers to discover how to build and manage open source software for the AIX operating system

 View Only
Expand all | Collapse all

Postgresql - Error with create vectorscale index: access method "diskann" does not exist

  • 1.  Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 16 days ago

    Hi,

    after resolving issues with Rust and libclang during installation of pgvectorscale on AIX I've got troubles when creating an index with the vectorscale access method.

    There were no errors while installing pgvectorscale but some warnings. But they should not harm. And I've could create vectorscale extension without any errors.

    postgres=# \dx
                     List of installed extensions
      Name   | Version |   Schema   |         Description
    ---------+---------+------------+------------------------------
     plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
    (1 row)

    postgres=# create extension vector;
    CREATE EXTENSION
    postgres=# \dx
                                 List of installed extensions
      Name   | Version |   Schema   |                     Description
    ---------+---------+------------+------------------------------------------------------
     plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
     vector  | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods
    (2 rows)

    postgres=# create extension vectorscale;
    CREATE EXTENSION
    postgres=# \dx
                                   List of installed extensions
        Name     | Version |   Schema   |                     Description
    -------------+---------+------------+------------------------------------------------------
     plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
     vector      | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods
     vectorscale | 0.7.1   | public     | diskann access method for vector search
    (3 rows)

    postgres=# CREATE TABLE if not exists test (
        id int PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY,
        embedding VECTOR(3)
    );
    CREATE TABLE
    postgres=# insert into test (embedding) values ('[1,2,3]'), ('[4,5,6]');
    INSERT 0 2
    postgres=# CREATE INDEX ON test USING diskann (embedding vector_cosine_ops);
    ERROR:  access method "diskann" does not exist

    Bizarrely, I cannot find the diskann access method in pg_am. The access methods of vector (hnsw and ivfflat) are here.

    postgres=# select oid from pg_proc where proname like 'diskann%';
     oid
    -----
    (0 rows)

    postgres=# select * from pg_am;
      oid  | amname  |      amhandler       | amtype
    -------+---------+----------------------+--------
         2 | heap    | heap_tableam_handler | t
       403 | btree   | bthandler            | i
       405 | hash    | hashhandler          | i
       783 | gist    | gisthandler          | i
      2742 | gin     | ginhandler           | i
      4000 | spgist  | spghandler           | i
      3580 | brin    | brinhandler          | i
     50292 | ivfflat | ivfflathandler       | i
     50294 | hnsw    | hnswhandler          | i
    (9 rows)

    postgres=# select oid, proname from pg_proc where proname like 'hnsw%' or proname like 'ivfflat%';
      oid  |         proname
    -------+-------------------------
     50291 | ivfflathandler
     50293 | hnswhandler
     50295 | ivfflat_halfvec_support
     50296 | ivfflat_bit_support
     50297 | hnsw_halfvec_support
     50298 | hnsw_bit_support
     50299 | hnsw_sparsevec_support
    (7 rows)

    postgres=# select * from pg_am where amname = 'diskann';
     oid | amname | amhandler | amtype
    -----+--------+-----------+--------
    (0 rows)

    What's going on here? Where is the fault?

    Is the installation of pgvectorscale incomplete? I haven't seen any errors during installation.

    By the way, after installation of pgvectorscale the library libvectorscale.a was installed in /opt/freeware/lib64/postgresql. But I assume the includes shared-object is required in this directory an I've extracted it.

    /opt/freeware/lib64/postgresql# ar -X64 -x libvectorscale.a libvectorscale.so

    I've restarted postgres, dropped and recreated the extensions. But it doesn't help either.

    Any ideas?



    ------------------------------
    Emanuel Reisinger
    ------------------------------


  • 2.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 15 days ago

    HI Emanuel

    Can you try to enable pg_diskann as suggested in https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-use-pgdiskann .

    Thanks
    Ranjit



    ------------------------------
    Ranjit Ranjan
    ------------------------------



  • 3.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 15 days ago

    Hi Ranjit,

    I've tried it but it didn't work. As expected this extension is not available because I've compiled vectorscale which should include diskann as access method.

    postgres=# create extension pg_diskann;
    ERROR:  extension "pg_diskann" is not available
    DETAIL:  Could not open extension control file "/opt/freeware/share/postgresql/extension/pg_diskann.control": No such file or directory.
    HINT:  The extension must first be installed on the system where PostgreSQL is running.

    Regards



    ------------------------------
    Emanuel Reisinger
    ------------------------------



  • 4.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 15 days ago

    okay. 
    if you have complete build log of pgvectorscale , please attach it.
    Can you figure out where vectorscale.control is installed as a part of pgvectorscale and share the content of it. 

    Thanks
    Ranjit



    ------------------------------
    Ranjit Ranjan
    ------------------------------



  • 5.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 15 days ago

    attached the files.

    # cat /opt/freeware/share/postgresql/extension/vectorscale.control

    comment = 'diskann access method for vector search'
    default_version = '0.7.1'
    #module_pathname = '$libdir/pgvectorscale'
    relocatable = false
    superuser = true
    requires = 'vector'

    Regards



    ------------------------------
    Emanuel Reisinger
    ------------------------------

    Attachment(s)

    log
    pgvectorscale_build.log   32 KB 1 version
    sh
    pgvectorscale_build.sh   1 KB 1 version


  • 6.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 14 days ago

    Thanks Emanuel for the logs.
    I see from below code that,  using pg_extern macro which comes from pgrx , code is trying to create diskann method entry and handler for it. That means it's either not hitting or some other issues.
    https://github.com/timescale/pgvectorscale/blob/main/pgvectorscale/src/access_method/mod.rs

    Let me build and try once . 

    @Xing Xue
    How I can get access to libpgbackend.a . 

    Thanks
    Ranjit



    ------------------------------
    Ranjit Ranjan
    ------------------------------



  • 7.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 14 days ago

    Hi @Ranjit Ranjan, Shared libpgbackend.a with you through OneDrive.



    ------------------------------
    ------------------------------
    Xing Xue
    ------------------------------
    ------------------------------



  • 8.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 13 days ago
    Edited by Ranjit Ranjan 13 days ago

    Hi Emanuel,

    In AIX, by default, we don't support linux style shared object version system, but cargo-pgrx is not aware of it and it's creating SQL file considering version system .

    So you will see below entry in /opt/freeware/share/postgresql/extension/vectorscale--0.7.0--0.7.1.sql 

    CREATE OR REPLACE FUNCTION diskann_amhandler(internal) RETURNS index_am_handler PARALLEL SAFE IMMUTABLE STRICT COST 0.0001 LANGUAGE c AS '$libdir/vectorscale-0.7.1', 'amhandler_wrapper';

    For time being, I was able to to create diskann method by following below steps.  But while creating index , it's causing Postgres server to crash with illegal instruction. 

    • cp /opt/freeware/share/postgresql/extension/vectorscale--0.7.0--0.7.1.sql  /opt/freeware/share/postgresql/extension/vectorscale--0.7.1.sql
    • Make below changes. Mainly I am using absolute path of libvectorscale.so instead of '$libdir/vectorscale-0.7.1

           diff -u vectorscale--0.7.0--0.7.1.sql vectorscale--0.7.1.sql

    --- vectorscale--0.7.0--0.7.1.sql       2025-07-04 05:17:46.973462748 -0500

    +++ vectorscale--0.7.1.sql      2025-07-04 06:45:50.406922667 -0500

    @@ -10,7 +10,7 @@

     -- pgvectorscale/src/access_method/mod.rs:38

     -- vectorscale::access_method::amhandler

     

    -    CREATE OR REPLACE FUNCTION diskann_amhandler(internal) RETURNS index_am_handler PARALLEL SAFE IMMUTABLE STRICT COST 0.0001 LANGUAGE c AS '$libdir/vectorscale-0.7.1', 'amhandler_wrapper';

    +    CREATE OR REPLACE FUNCTION diskann_amhandler(internal) RETURNS index_am_handler PARALLEL SAFE IMMUTABLE STRICT COST 0.0001 LANGUAGE c AS '/opt/freeware/lib64/postgresql/libvectorscale', 'amhandler_wrapper';

     

         DO $$

         DECLARE

    @@ -34,7 +34,7 @@

     CREATE OR REPLACE FUNCTION "distance_type_cosine"() RETURNS smallint /* i16 */

     IMMUTABLE STRICT PARALLEL SAFE

     LANGUAGE c /* Rust */

    -AS '$libdir/vectorscale-0.7.1', 'distance_type_cosine_wrapper';

    +AS '/opt/freeware/lib64/postgresql/libvectorscale', 'distance_type_cosine_wrapper';

     /* </end connected objects> */

     

     /* <begin connected objects> */

    @@ -43,7 +43,7 @@

     CREATE OR REPLACE FUNCTION "distance_type_inner_product"() RETURNS smallint /* i16 */

     IMMUTABLE STRICT PARALLEL SAFE

     LANGUAGE c /* Rust */

    -AS '$libdir/vectorscale-0.7.1', 'distance_type_inner_product_wrapper';

    +AS '/opt/freeware/lib64/postgresql/libvectorscale', 'distance_type_inner_product_wrapper';

     /* </end connected objects> */

     

     /* <begin connected objects> */

    @@ -52,7 +52,7 @@

     CREATE OR REPLACE FUNCTION "distance_type_l2"() RETURNS smallint /* i16 */

     IMMUTABLE STRICT PARALLEL SAFE

     LANGUAGE c /* Rust */

    -AS '$libdir/vectorscale-0.7.1', 'distance_type_l2_wrapper';

    +AS '/opt/freeware/lib64/postgresql/libvectorscale', 'distance_type_l2_wrapper';

     /* </end connected objects> */

     

     /* <begin connected objects> */

    @@ -64,7 +64,7 @@

     ) RETURNS bool /* bool */

     IMMUTABLE STRICT PARALLEL SAFE

     LANGUAGE c /* Rust */

    -AS '$libdir/vectorscale-0.7.1', 'smallint_array_overlap_wrapper';

    +AS '/opt/freeware/lib64/postgresql/libvectorscale', 'smallint_array_overlap_wrapper';

     /* </end connected objects> */

     

     /* <begin connected objects> */

    •  After that you can restart the server and try to create DiskANN extension method.

    mydb3=# \dx

                                   List of installed extensions

        Name     | Version |   Schema   |                     Description                      

    -------------+---------+------------+------------------------------------------------------

     plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language

     vector      | 0.8.0   | public     | vector data type and ivfflat and hnsw access methods

     vectorscale | 0.7.1   | public     | diskann access method for vector search

    (3 rows)

    mydb3=# select * from pg_am where amname = 'diskann';

      oid  | amname  |     amhandler     | amtype 

    -------+---------+-------------------+--------

     50814 | diskann | diskann_amhandler | i

    (1 row)

    mydb3=# select * from pg_am;

      oid  | amname  |      amhandler       | amtype 

    -------+---------+----------------------+--------

         2 | heap    | heap_tableam_handler | t

       403 | btree   | bthandler            | i

       405 | hash    | hashhandler          | i

       783 | gist    | gisthandler          | i

      2742 | gin     | ginhandler           | i

      4000 | spgist  | spghandler           | i

      3580 | brin    | brinhandler          | i

     34492 | ivfflat | ivfflathandler       | i

     34494 | hnsw    | hnswhandler          | i

     50814 | diskann | diskann_amhandler    | i

    (10 rows)

    mydb3=# SELECT * FROM pg_catalog.pg_opclass c WHERE c.opcname = 'vector_cosine_ops';

      oid  | opcmethod |      opcname      | opcnamespace | opcowner | opcfamily | opcintype | opcdefault | opckeytype 

    -------+-----------+-------------------+--------------+----------+-----------+-----------+------------+------------

     34520 |     34492 | vector_cosine_ops |         2200 |       10 |     34519 |     34430 | f          |          0

     34535 |     34494 | vector_cosine_ops |         2200 |       10 |     34534 |     34430 | f          |          0

     50820 |     50814 | vector_cosine_ops |         2200 |       10 |     50819 |     34430 | t          |          0

    (3 rows)

    mydb3=# CREATE INDEX ON test USING diskann (embedding vector_cosine_ops);
    server closed the connection unexpectedly
            This probably means the server terminated abnormally
            before or while processing the request.
    The connection to the server was lost. Attempting reset: Succeeded.
    mydb3=# 

    I don't see any use full info from core file as server is not compiled for debugging. Try in your system and let us know the result. 

    Thanks

    Ranjit



    ------------------------------
    Ranjit Ranjan
    ------------------------------



  • 9.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 13 days ago

    Same here. Same error.

    on debug5 log level I don't get any helpful information.

    2025-07-04 16:07:09.442 DFT [17760736] LOG:  parameter "log_min_messages" changed to "debug5"
    2025-07-04 16:07:09.442 DFT [17760736] DEBUG:  sending signal 1 to process 13435226
    2025-07-04 16:07:09.446 DFT [13435226] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:09.446 DFT [13435226] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:09.449 DFT [15073540] DEBUG:  checkpointer updated shared memory configuration values
    2025-07-04 16:07:09.451 DFT [13959498] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:09.451 DFT [13959498] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:09.451 DFT [13959498] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:09.451 DFT [13959498] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:15.992 DFT [18809194] DEBUG:  InitPostgres
    2025-07-04 16:07:15.992 DFT [18809194] DEBUG:  my backend ID is 3
    2025-07-04 16:07:15.993 DFT [18809194] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:16.001 DFT [17760736] DEBUG:  forked new backend, pid=18809194 socket=10
    2025-07-04 16:07:16.004 DFT [18809194] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:19.595 DFT [18809194] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:19.600 DFT [18809194] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:31.132 DFT [18809194] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:31.198 DFT [18546978] DEBUG:  InitPostgres
    2025-07-04 16:07:31.199 DFT [18546978] DEBUG:  my backend ID is 4
    2025-07-04 16:07:31.199 DFT [18546978] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:31.202 DFT [17760736] DEBUG:  forked new backend, pid=18546978 socket=10
    2025-07-04 16:07:31.202 DFT [18546978] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
    2025-07-04 16:07:31.778 DFT [17760736] DEBUG:  reaping dead processes
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  server process (PID 18809194) was terminated by signal 4: Illegal instruction
    2025-07-04 16:07:31.779 DFT [17760736] DETAIL:  Failed process was running: CREATE INDEX ON test USING diskann (embedding vector_cosine_ops);
    2025-07-04 16:07:31.779 DFT [17760736] LOG:  server process (PID 18809194) was terminated by signal 4: Illegal instruction
    2025-07-04 16:07:31.779 DFT [17760736] DETAIL:  Failed process was running: CREATE INDEX ON test USING diskann (embedding vector_cosine_ops);
    2025-07-04 16:07:31.779 DFT [17760736] LOG:  terminating any other active server processes
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 13435226
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 18546978
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 17236248
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 15073540
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 7078236
    2025-07-04 16:07:31.779 DFT [17760736] DEBUG:  sending SIGQUIT to process 13959498
    2025-07-04 16:07:31.803 DFT [17760736] DEBUG:  reaping dead processes
    2025-07-04 16:07:31.826 DFT [17760736] DEBUG:  reaping dead processes
    2025-07-04 16:07:31.868 DFT [17760736] DEBUG:  reaping dead processes
    2025-07-04 16:07:31.891 DFT [17760736] DEBUG:  reaping dead processes
    2025-07-04 16:07:31.891 DFT [17760736] DEBUG:  server process (PID 18546978) exited with exit code 2

    It sadly seems that pgvectorscale is not built to run on AIX. All these problems are no good start to use pgvectorscale on productive systems on AIX.



    ------------------------------
    Emanuel Reisinger
    ------------------------------



  • 10.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 9 days ago

    I will try to debug further on this with help from compiler team.  
    What about pgvector methods ? Are you able to test it out on AIX . is it working as expected ?

    Thanks
    Ranjit



    ------------------------------
    Ranjit Ranjan
    ------------------------------



  • 11.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 9 days ago

    The access methods (indices) of pgvector are working as expected.

    Easy and without any errors to install and working at the first attempt.

    Distinction:

    • The methods of pgvector are for table entries less than 1M and are held in memory.
    • pgvectorscales method is for table entries more than 1M and is optimized for search on disk.

    At the moment our entries will have less than 500K entries. So we can use pgvector.

    But in future scenarios we will have more than 1M entries and we will have to use pgvectorscale for that.

    Thanks for your support and effort. Regards



    ------------------------------
    Emanuel Reisinger
    ------------------------------



  • 12.  RE: Postgresql - Error with create vectorscale index: access method "diskann" does not exist

    Posted 13 days ago

    A little progress from our side.

    On Linux there is a shared-object called vectorscale-0.7.1-dev.so in pg_config --pkglibdir. On AIX there is only libvectorscale.a(libvectorscale.so) in that directory. Extracting libvectorscale.so from archive does not help (drop/create extension as well).

    But if I create a link ln -s libvectorscale.so vectorscale-0.7.1-dev.so and drop/create the extension, then I get the diskman_amhandler in pg_am but when I connect to a database the handler is not available (not found in pg_am).

    I've also noticed that die pg_am.oid of the vector extension handlers ivfflat and hnsw are different if I'm not connected and connected to a database. But all other handlers have the same oid.

    Has anyone any ideas?

    Regards,



    ------------------------------
    Emanuel Reisinger
    ------------------------------